Information Technology: Better Informed Decision Making Needed on Navy's Next Generation Enterprise Network Acquisition
Published: 2011-03-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of the Navy (DON), a major component of the Department of Defense (DOD), has launched its Next Generation Enterprise Network (NGEN) program to replace the Navy Marine Corps Intranet (NMCI) program. NGEN capabilities, such as secure transport of voice and data, data storage, and e-mail, are to be incrementally acquired through multiple providers. As planned, the first increment is expected to provide comparable NMCI capabilities, additional information assurance, and greater DON network control, at a cost of about $50 billion through fiscal year 2025. Given the size, importance, and complexity of NGEN, GAO was asked to determine whether DON has sufficiently analyzed alternative acquisition approaches and has a reliable schedule for executing the program, and whether program acquisition decisions have been performance- and risk-based. To do this, GAO reviewed the NGEN analysis of alternatives, integrated master schedule, and key milestone decisions.
section_title: What GAO Found
paragraphs: DON did not sufficiently analyze alternative acquisition approaches for NGEN because the alternatives analysis contained key weaknesses, and none of the alternatives assessed match the current acquisition approach. Specifically, the cost estimates for the respective alternatives were not reliable because they were not substantially accurate, and they were neither comprehensive nor credible. Further, the operational effectiveness analysis, the other key aspect of an analysis of alternatives, did not establish and analyze sufficient measures for assessing each alternative's ability to achieve program goals and deliver program capabilities. Moreover, the acquisition approach that DON is actually pursuing was not one of the alternatives assessed in the analysis, and it is riskier and potentially costlier than the alternatives analyzed because it includes a higher number of contractual relationships. According to program officials, the analysis reflects the most that could be done in the time that was available to complete it, and they do not view the alternative selected as materially different from the assessed alternatives, even though it is about $4.7 billion more costly. DON does not have a reliable schedule for executing NGEN. Only two of the four subschedules that GAO reviewed, each of which help form the master schedule, adequately satisfied any of the nine practices that are associated with developing and maintaining a reliable schedule. These weaknesses have contributed to delays in key program milestones. During the course of GAO's review, DON stated that action was taken to address some, but not all, of these weaknesses. According to program officials, schedule estimating was constrained by staffing limitations. NGEN acquisition decisions were not always performance- and risk-based. In particular, the program was approved in the face of known performance shortfalls and risks. For example, the program was approved at a key acquisition review despite the lack of defined requirements, which was recognized as a risk that would impact the completion of other key documents, such as the test plan. This risk was later realized as a critical issue. According to program officials, the decisions to proceed were based on their view that they had sufficiently mitigated known risks and issues. Collectively, these weaknesses mean that DON does not have a sufficient basis for knowing that it is pursuing the best approach for acquiring NGEN capabilities and the program's cost and schedule performance is unlikely to track to estimates.
section_title: What GAO Recommends
paragraphs: GAO is recommending that DOD limit further investment in NGEN until it conducts an interim review to reconsider the selected acquisition approach and addresses issues discussed in this report. In its comments, DOD stated that it did not concur with the recommendation to reconsider its acquisition approach; GAO maintains that without doing so, DOD cannot be sure it is pursuing the most cost-effective approach.

section_title: Letter
paragraphs: The Department of the Navy (DON) established the Next Generation  Enterprise Network (NGEN) program in 2007 to replace the Navy Marine  Corps Intranet (NMCI), which is provided through a DON-wide network  services contract. As envisioned, NGEN capabilities, including secure  transport of voice and data, data storage, and e-mail, are to be  incrementally acquired through multiple providers. The first increment is  to include capabilities comparable to NMCI, as well as enhanced  information assurance and increased government control over network  operations. To date, NGEN has reportedly spent about $432 million on  work associated with the transition from NMCI. The first increment is to  be fully operational in March 2014 and is to cost approximately $50 billion  through 2025.
As agreed, our objectives were to determine whether (1) DON has  sufficiently analyzed alternative approaches for acquiring NGEN, (2) DON  has a reliable schedule for executing NGEN, and (3) acquisition decisions  have been performance- and risk-based. To accomplish our objectives, we  analyzed the NGEN alternatives analysis report and underlying support,  the program’s integrated master schedule, program performance  assessments and risk reports, and executive acquisition decision briefings  and meeting minutes, among other things. We conducted this performance  audit from October 2009 to February 2011, in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives. Additional details on our objectives, scope, and methodology  are discussed in appendix I.
subsections: section_title: Background
paragraphs: DON is a major component of the Department of Defense (DOD),  consisting of the Navy and the Marine Corps. It is a large and complex  organization, whose primary mission is to organize, train, maintain, and  equip combat-ready naval forces capable of winning wars, deterring  aggression by would-be foes, preserving freedom of the seas, and  promoting peace and security. To support this mission, DON performs a  variety of interrelated and interdependent information technology (IT)- dependent functions. In fiscal year 2010, DON’s IT budget was  approximately $7.4 billion, for 971 investments. NGEN is one such system  investment.
subsections: section_title: Overview of NGEN
paragraphs: NGEN is to provide secure data and IT services, such as data storage, e- mail, and video-teleconferencing, to the Navy and the Marine Corps.  NGEN is also intended to provide the foundation for DON’s future Naval  Networking Environment. DON is acquiring NGEN through multiple  providers (contractors) to replace and improve the enterprise network and  services provided by NMCI. It is to be developed incrementally, with the  first increment to provide comparable NMCI capabilities, additional  information assurance, and increased government control of the network.  Future increments have yet to be defined. The program’s preliminary life  cycle cost estimate (through fiscal year 2025) for the first increment is  about $50 billion. As of September 30, 2010, the NGEN program had  reportedly spent about $432 million.
To bridge the time frame between the end of the NMCI contract and the  full transition to NGEN, DON awarded a $3.7 billion continuity of services  contract in July 2010 to the NMCI service provider, Hewlett Packard  Enterprise Services. In addition to providing continuity of network  services, the contract includes transition services and transfer to DON of  NMCI infrastructure and intellectual property, as the NGEN contracts are  to require use of the NMCI infrastructure and access to processes,  procedures, and technical data. The continuity of services contract is  scheduled to run from October 2010 through April 2014.
To reduce the risk during the transition period from NMCI to NGEN, DON  is currently performing eight early transition activities. The activities are  discrete efforts intended to establish government management  capabilities, allow for greater participation in operational decisions, and  help expedite the transition time. Table 1 describes each of these  activities.
To deliver NGEN capabilities, DON plans to award five contracts. See  table 2 for a description of these contracts.
According to the NGEN Acquisition Strategy, DON plans to complete the  Marine Corps’ initial transition to NGEN in January 2012 and final  transition in February 2013. The Navy’s initial and final transition to NGEN  are scheduled to be completed in December 2012 and March 2014,  respectively.
section_title: NGEN Oversight and Management Roles and Responsibilities
paragraphs: To manage the acquisition and deployment of NGEN, DON established a  program management office within the Program Executive Office for  Enterprise Information Systems. The program office manages the  program’s cost, schedule, and performance and is responsible for ensuring  that the program meets its objectives. In addition, various DOD and DON  organizations share program oversight and review responsibilities. Table 3  lists key entities and their roles and responsibilities.
section_title: Overview of DOD and DON Acquisition Processes and NGEN Status
paragraphs: NGEN is subject to both Office of the Secretary of Defense (OSD) and  DON Major Automated Information System (MAIS) acquisition policy and  guidance, which require it to comply with Defense Acquisition System  (DAS) requirements. According to these requirements, all MAIS programs  require a Materiel Development Decision prior to entering the first DAS  phase. In making this decision, the milestone decision authority is to  review the Initial Capabilities Document, which defines operational goals  and needed capabilities, and authorizes the phase in which the program is  to enter the DAS. The system consists of five key program life cycle phases  and three related milestone decision points. Table 4 provides a description  of each DAS phase.
In addition to Defense Acquisition System requirements, according to DON  guidance and policy, all DON MAIS and pre-MAIS programs are required to  go through a “Two-Pass/Six-Gate” acquisition review process. The first  pass, which consists of Gates 1 through 3, is focused on requirements  development and validation and is led by the Chief of Naval Operations or  the Commandant of the Marine Corps. The second pass, which consists of  Gates 4 through 6, is focused on developing and delivering a solution via  systems engineering and acquisition and is led by the Assistant Secretary  of the Navy (Research, Development, and Acquisition). In addition to  meeting specific criteria for passing a given gate and proceeding to the  next gate, all gate reviews are to consider program health (i.e., satisfactory  cost and schedule performance, known risks, and budget adequacy) in  deciding whether to proceed. Table 5 lists the key purpose of each gate  review.
The DAS and DON acquisition phases and decision points for MAIS  programs are illustrated in figure 1.
As depicted in figure 2, DON completed a Gate 3 review of NGEN  requirements in April 2008. In April 2009, the DON CIO completed the  AOA for NGEN increment 1, and at the Gate 2 review the same month, the  Deputy Chief of Naval Operations (Integration of Capabilities and  Resources) and the Deputy Marine Corps Commandant for Programs and  Resources approved the AOA to be submitted to the NGEN AOA Advisory  Group. The advisory group subsequently approved the analysis and  forwarded it in April 2009 to OSD Cost Assessment and Program  Evaluation (CAPE), which approved it in December 2009. DON conducted  a Gate 4 review of its System Design Specification in November 2009, and  a Gate 5 review of its Transport Services request for proposal in October  2010. DON plans to conduct a Gate 6 review in July 2011.
In May 2010, the USD (AT&L) completed the NGEN Materiel Development  Decision, which designated the first increment of NGEN as a MAIS and  authorized the program to enter the DAS in the production and  deployment phase. A Milestone C review is currently planned for August  2011. In June 2010, the USD (AT&L) approved the current acquisition  approach.
section_title: Current NGEN Acquisition Approach Is Not Grounded in a Reliable AOA
paragraphs: An AOA is intended to help identify the most promising acquisition  approach by comparing alternative solutions’ costs and operational  effectiveness. The NGEN AOA contained key weaknesses in its cost  estimates and operational effectiveness analysis that impaired its ability to  inform investment decision making. Further, none of the alternatives in  this analysis match the current acquisition approach, and these differences  have not been analyzed to determine the breadth of risk that exists.  According to DON officials, the AOA reflects the most that could be  accomplished in the time available to meet an imposed deadline. In  addition, OSD officials stated that the differences between the current  approach and the alternatives that were assessed are, in their view, not  significant. However, the current approach is estimated to cost at least  $4.7 billion more than any of the AOA alternatives. Without sufficient  information to understand the differences in the relative costs and  operational effectiveness among alternatives, decision makers lack  assurance that their selected approach is the most promising and cost- effective course of action.
subsections: section_title: NGEN AOA Was Not Based on Reliable Cost Estimates
paragraphs: According to relevant guidance, a key component of an AOA is a cost  analysis that provides for cost estimates of each alternative. As such, cost  estimates should be reliable in order to provide the basis for informed  investment decision making, realistic budget formulation, meaningful  progress measurement, and accountability for results. Our research has  identified four characteristics of a high-quality, reliable cost estimate: well- documented, comprehensive, accurate, and credible.
The NGEN AOA assessed four alternatives. All alternatives were  assumed to deliver the same NMCI capabilities and the technolog alternatives was assumed to be substantially the same. The primary  differences among the alternatives were how NGEN was to be acquired,  managed, and operated. Table 6 below provides a description of each  alternative.
The four alternatives’ estimated costs for increment 1 from fiscal year 2011  to fiscal year 2015 ranged from $10.25 billion (alternative 1) to $10.84  billion (alternatives 2 and 3V). However, the estimates were not reliable  because they substantially met only one of the characteristics of reliable  cost estimates. Specifically,    The AOA cost estimates were substantially well-documented. To be well- documented, the cost estimates should state the purpose of the estimate;  provide program background, including a system description; provide a  schedule for developing the estimates; specify the scope of the estimate  (in terms of time and what is and is not included); disclose key ground  rules and assumptions, data sources, calculations performed and their  results, the estimating methodology and rationale, and the results of a risk  analysis; and provide a conclusion about whether the cost estimate is  reasonable. Moreover, this information should be captured in such a way  that the data used to derive the estimate can be traced to their sources.  Finally, the cost estimates should be reviewed and accepted by  management.
Although the AOA did not sufficiently document the schedule, scope, and  results of the risk analysis, it defined the purpose of the estimate; provided  program background (e.g., system description); and disclosed ground  rules and assumptions, data sources, calculations performed and their  results, and the estimating methodology. Also, the data used to derive the  estimates were captured in such a way that they could largely be traced to  their sources, and the final AOA was reviewed and accepted by DON and  OSD oversight entities.
The AOA cost estimates were not comprehensive. To be comprehensive,  the cost estimates should include all government and contractor costs  over the program’s full life cycle, from program inception through design,  development, deployment, and operation and maintenance to retirement.  They should also provide sufficient detail and reflect all cost-influencing  ground rules and assumptions.
However, the cost estimates were not full life cycle costs. Instead, they  only included government and contractor costs for a 5-year period from  fiscal year 2011 to fiscal year 2015, covering 2 years of continued NMCI  services with the current provider, 2 years of transition to the new  provider(s), and 1 year of NGEN operation and maintenance. DON and  OSD CAPE officials attributed this to the assumption that NGEN  increment 1 contracts would have a 5-year period of performance and that  future NGEN increments might be introduced after that period.
Further, while the estimates were based on a cost element structure that  was decomposed to a sufficient level of detail, and the documentation  largely identified ground rules and assumptions, the cost estimates did not  reflect all assumptions identified in the AOA, such as schedule and  performance risks associated with (1) implementing IT processes, (2)  expanding the government workforce, and (3) formulating the NGEN  contracts. These were significant cost-influencing risks and thus should  have been incorporated into the estimates.
The AOA cost estimates were not substantially accurate. To be accurate,  the cost estimates should not be overly conservative or optimistic and  should be, among other things, based on an assessment of the most likely  costs, and adjusted properly for inflation. In addition, steps should be  taken to minimize mathematical mistakes and to ground the estimate in  documented assumptions that can be verified by supporting data and a  historical record of actual cost and schedule experiences on comparable  programs.
To DON’s credit, the cost estimates were developed based on NMCI  historical cost data, were adjusted properly for inflation, contained few  mathematical mistakes, and were largely grounded in documented  assumptions. However, the supporting data for key assumptions were not  verified. For example, all estimates assumed that transition activity costs  would amount to about 18 percent of the estimated cost of NGEN in its  first year of operation, and alternative 3’s estimate assumed that total cost  would be reduced by 10 percent due to increased competition from its  multicontract approach. However, the supporting data used by Deloitte  Consulting for these assumptions were not provided to DON or the  independent government review team for verification because the data  were proprietary to the contractor. Further, NMCI historical data were  only available at an aggregate level, so the team had to rely on subject- matter experts and other sources to estimate costs at a finer level of detail.
The AOA cost estimates were not credible. To be credible, the cost  estimates should discuss any limitations in the analysis due to uncertainty  or biases surrounding the data and assumptions. Major assumptions  should be varied and other outcomes computed to determine how  sensitive the estimate is to changes in the assumptions. Risk and  uncertainty inherent in the estimate should be assessed and disclosed.  Further, the estimate should be properly verified by, for example,  comparing the results with an independent cost assessment.
While the AOA identified limitations in the cost analysis, such as the use of  NMCI data that did not reflect prices of other service providers, and  evaluated the impact on costs of using different transition timeline  scenarios, it did not include a sensitivity analysis of the key cost driver  (i.e., the number of personnel needed to manage NGEN), despite concerns  that the Navy’s estimates of these numbers were not stabilized at the time  of the AOA. In addition, while each cost estimate included a cost risk  analysis based on the quality of data used, there were discrepancies in  how the analysis was conducted and reported. For example, the cost for  local area network facilities was estimated based on the contractor’s  experience, which was considered by the cost team to be a less credible  data source, but it was scored higher on their risk scale, indicating that the  data source was more credible. Also, schedule and performance risks  were not quantified and reflected in the estimates, which is significant  because a qualitative assessment of schedule and performance risks  among alternatives revealed increased risk in implementing a segmented  approach. If such risks had been quantified and reflected in the estimates,  the results would have shown higher costs associated with alternatives 3  and 3V. Nevertheless, the AOA concluded that there was no significant  cost difference among the alternatives.
In addition, the cost estimates were not validated by the independent team  responsible for reviewing the cost analysis. Specifically, independent  review team officials told us that they participated in a line-by-line review  of the cost model where they raised comments and questions to the cost  team. However, about 69 percent of the team’s comments were not fully  addressed and included notable concerns, such as the questionable use of  certain industry-based assumptions that may not be comparable to a  program as large as NGEN or to the government environment.  Independent review team officials attributed the comments not being  closed to the fact that the team did not have authority over the cost model  to ensure that its comments were addressed. Further, these officials told  us that they were not asked to review the final version of the cost model,  which was the version that first introduced alternative 3V, and their review  of the final version of the AOA report occurred after the DON CIO had  submitted it to OSD CAPE for final approval.
According to officials responsible for developing the AOA, the weaknesses  in the AOA cost estimates largely exist because there was not enough time  to conduct a more thorough analysis. Specifically, they told us that the  AOA schedule was constrained because the program wanted to get  requests for proposals for NGEN contracts out by a predetermined date.  This position was also supported by various management meeting minutes  and other artifacts that we reviewed. However, DOD and DON officials  disagreed with this position and told us that the time allotted to conduct  the AOA did not negatively impact its quality or scope. A time-constrained  approach is not consistent with DOD guidance, which states that the scope  of an alternatives analysis should be proportionate to the amount of  resources affected by the decision, with more significant programs  receiving more analytical attention. The combination of the AOA  weaknesses we identified, and the fact that NGEN has a preliminary  program life cycle cost estimate of about $50 billion for increment 1 and is  intended to provide the foundation for DON’s future networking  environment, suggest the need for considerable analytical attention to  alternative approaches.
Without reliable cost estimates for each alternative, decision makers did  not have a sound basis for making an informed decision on an NGEN  solution. Most notably, since the estimates did not reflect increased risks  associated with the segmented approach, the differences in the  alternatives’ costs were understated, and the amount of risk and costs  accepted by proceeding with a segmented approach were not fully  understood.
section_title: AOA Did Not Sufficiently Assess Operational Effectiveness
paragraphs: In addition to including reliable cost estimates, an AOA should assess how  well each alternative satisfies required capabilities or goals. According to  DOD guidance, such an analysis should (1) identify the operational  capabilities and goals to be achieved with the system solution, (2)  establish quantitative or qualitative measures for evaluating the  operational effectiveness of each alternative, and (3) assess the ability of  each alternative to achieve these measures.
While the AOA identified program capabilities and goals, it did not  sufficiently assess the alternatives’ operational effectiveness, making it  unclear how well the alternatives would actually satisfy NGEN capabilities  and goals. Specifically,    The AOA identified capabilities and goals that the system solution should  achieve. Among other things, these included addressing NMCI capability  limitations identified based on 8 years of operational experience, as well  as capabilities needed to support DOD and DON networking strategies for  DOD’s Global Information Grid Network Operations and DON’s future  Naval Networking Environment. (See table 8 for these capabilities and  goals.)
The AOA did not establish quantitative or qualitative measures for  assessing the alternatives’ ability to achieve the identified NGEN  capabilities and goals, as shown in table 8. For example, one of the  capabilities was visibility into root causes for major network outages,  which the AOA merely concluded that alternatives 2, 3V, and 3 were  equally effective in addressing, even though no quantitative or qualitative  measures of the alternatives’ respective ability to provide visibility into  root causes were defined. Further, the AOA did not discuss the  methodology for assessing the alternatives. Rather, it simply states that it  was a qualitative assessment.
While the AOA did not establish measures for assessing the alternatives’  ability to achieve NGEN capabilities and goals, it did establish several  quantitative measures to differentiate among the alternatives’ respective  approaches to acquiring, managing, and delivering NMCI capabilities.  However, these measures alone do not provide insight into how they  would influence the operational effectiveness of each alternative because  they were not linked to NGEN capabilities and goals, and they did not  provide sufficient insight for selecting a preferred alternative. For  example, while the AOA recognized that an increase in the number of  contractual relationships would result in more complexity and risk in  implementing the alternative, it did not include measures for quantifying  how much more risk is introduced as the number of contractual  relationships increases. (See table 7 for the measures that were provided  in the AOA.)
In addition, the AOA included a separate assessment of the likelihood of  each alternative to successfully implement IT best practices for end-to-end  IT service delivery (i.e., IT Service Management framework). To DON’s  credit, the approach used to measure the alternatives in this assessment  was more structured and better documented. Specifically, the AOA team  conducted table-top exercises with subject-matter experts representing  each of the communities that will contribute to the acquisition, operation,  and oversight of NGEN, and it worked through scenarios, such as  everyday operations and responding to a computer network incident, to  determine the extent to which each alternative could employ IT best  practices to address a given scenario. The team captured comments made  by participants and used them to infer rankings that resulted in numerical  scores for each alternative.
The AOA did not assess the alternatives’ ability to address capabilities and  goals using defined measures of operational effectiveness because, as  stated previously, no measures were established. Instead, it compared the  alternatives based on qualitative determinations of whether the capability  or goal was either met or partially met. (See table 8 for the results of  DON’s assessment.)
As with the cost estimates, officials responsible for developing the AOA  told us that the operational effectiveness analysis was subject to time  constraints so that requests for proposals could be issued on time.  Although DOD and DON officials told us that the time allotted to conduct  the AOA did not negatively impact its quality or scope, our review suggests  otherwise. Further, the time-constrained approach is not consistent with  DOD guidance, which states that the scope of an alternatives analysis  should be proportionate to the resources affected by the decision, with  more significant programs receiving more analytical attention.
Without a more thorough effectiveness analysis, decision makers did not  have a sound basis for making an informed decision on the best NGEN  alternative to pursue. Instead, DON has selected a segmented approach on  the basis that it would provide increased flexibility in meeting NGEN  capabilities and goals with no additional cost, even though the degree of  increased flexibility among the alternatives remains unclear.
section_title: Current Acquisition Approach Was Not Addressed in the AOA and Is Riskier and Costlier Than Other Approaches in the Analysis
paragraphs: According to DOD guidance, an AOA should examine viable solutions  with the goal of identifying the most promising option, thereby informing  acquisition decision making. However, the segmented approach currently  being pursued by DON was not one of the alternatives assessed in the  AOA. Specifically, the current approach has more contracts, a different  segmentation scheme, and a different transition timeline than the analyzed  alternatives. Further, the impact of these differences in terms of how they  compare to the original alternatives was not assessed.
The approach that is being pursued by the program office includes a  higher number of contracts than those analyzed in the AOA. Given that the  AOA highlighted greater schedule and performance risks as the number of  contracts and contractual relationships in the approach increase, the  relative schedule and performance risks for the current approach are  likely greater than those for alternative 3, and therefore are likely to result  in greater costs. In support of this likelihood, DON’s November 2009 risk- adjusted preliminary program life cycle cost estimate for the current  approach for fiscal year 2011 through fiscal year 2015 shows that the  current approach will cost at least an estimated $4.7 billion more than any  of the alternatives in the AOA. (See table 9 for a comparison of the current  approach to the approaches assessed in the AOA and fig. 3 for an  illustration of the contractual relationships associated with DON’s current  approach.)
OSD CAPE officials told us that they believe the differences between the  current approach and alternatives assessed in the AOA are not significant  because DON is still pursing a segmented approach and that the  differences were the result of “an appropriate evolution of the segmented  approach.” They further said that the increased risks in the current  approach are offset by mitigating factors, such as the use of staggered  phases to implement NGEN and the use of more efficient segmentation  schemes. However, we have yet to receive any analysis to support their  positions, and the current approach is estimated to cost about $4.7 billion  more. As a result, DON cannot demonstrate that it is pursuing the most  cost-effective approach for acquiring NGEN capabilities and meeting  NGEN goals.
section_title: NGEN’s Schedule Does Not Provide a Reliable Basis for Program Execution, and Program Delays Are Occurring
paragraphs: The success of a large-scale acquisition program depends in part on having  a reliable schedule that defines, among other things, when work activities  and milestone events will occur, how long they will take, and how they are  related to one another. As such, the schedule not only provides a road map  for systematic program execution but also provides the means by which to  gauge progress, identify and address potential problems, and promote  accountability. Without a reliable schedule, it is likely that established  program milestones will slip. In the case of NGEN, such delays are already  being experienced.
Our work has identified nine best practices associated with developing  and maintaining a reliable schedule. These are (1) capturing all activities,  (2) sequencing all activities, (3) assigning resources to all activities, (4)  establishing the duration of all activities, (5) integrating schedule activities  horizontally and vertically, (6) establishing the critical path for all  activities, (7) identifying reasonable “float” between activities, (8)  conducting a schedule risk analysis, and (9) updating the schedule using  logic and durations. See table 10 for a description of each of these best  practices.
In December 2009, NGEN established a baseline integrated master  schedule composed of over 25 separate underlying schedules (or  subschedules) to capture program milestones and the expected  completion dates for activities leading up to them. However, the most  current version of this schedule (May 2010) that was available at the time  we began our review was not reliable because only two of the four  subschedules that we analyzed substantially met any of the nine  practices. The results of our analysis of the four subschedules are  summarized in table 11.
Capturing all activities. All four subschedules partially met this practice.  Specifically, the majority of the activities contained in these subschedules  could be mapped back to the program’s NGEN work breakdown structure.  However, this structure is defined at a high level and is not expected to be  further decomposed into planned work products and deliverables until the  program enters the deployment phase when NGEN contracts are awarded.  Until this structure is sufficiently defined, it cannot be determined whether  the program schedules capture all work needed to accomplish program  objectives. For example, we identified risk mitigation activities for 10  active risks that should have been, but were not, captured as scheduled  work.
During our review, program officials told us that they had since taken  steps to ensure that all risk mitigation activities are added to the schedule.  However, until NGEN work is sufficiently defined, the program does not  have complete assurance that the activities currently captured in the  various schedules support NGEN increment 1.
Sequencing all activities. One subschedule substantially met this practice  while the other three minimally met it. The subschedule that substantially  met this practice had less than 1 percent of activities missing a  predecessor or successor dependency. Of the remaining three  subschedules, two did not identify predecessor or successor activities for  over half of the activities in their schedules. This is of concern because if  an activity that has no logical successor slips, the schedule will not reflect  the effect of these slips on the critical path, float, or scheduled start dates  of “downstream” (i.e., later) activities. Additionally, one subschedule had  “constraints” placed on about 73 percent of its activities, meaning that  these activities cannot begin earlier even if upstream work is completed  ahead of schedule. According to program officials, they are working to  reduce the number of constraints in the schedule. However, until activities  are properly sequenced, these issues reduce the credibility of the dates  calculated by the scheduling tool.
Assigning resources to all activities. Program officials told us that they  do not assign resources to any of the program schedules. They stated that  the effort necessary to assign resources within the schedules would be  significant and that they did not have the staff available to do this.  However, without proper allocation of resources in the schedule, the  program office cannot accurately forecast the likelihood that activities will  be completed by their projected end dates, and the risk that key  milestones will slip increases.
Establishing the duration of all activities. Two subschedules met this  practice while two only minimally met it. The two subschedules that met  this practice had established activities with reasonable durations—the  majority of which were under 30 days. The remaining two did not establish  reasonable durations for their activities. For example, the majority of the  activities that were in progress for the Transition Integrated Product Team  subschedule had durations ranging from 50 days to 1000 days. When such  long durations are assigned to work activities, it is likely that the activity is  not defined to the necessary level to identify all the work that must be  performed.
Integrating schedule activities vertically and horizontally. One of the  subschedules substantially met and the other three partially met this  practice. The subschedule that substantially met the practice is  horizontally aligned, meaning activities are logically sequenced, and  vertically aligned, meaning that detailed activities roll up into larger  summary tasks. The other three subschedules are also vertically aligned;  however, they are unable to demonstrate horizontal integration because,  as previously discussed, activities were not all logically sequenced.
The integration issues identified on these subschedules also impact the  NGEN master schedule. Because of the high number of missing  dependencies, the number of in-progress activities with durations  exceeding 30 days, and the high number of constraints, the master  schedule is likely not fully horizontally integrated.  Further, one of the subschedules is not vertically aligned with the master  schedule because none of the key work activities in the subschedule were  included in the master schedule. In addition, the master schedule was not  integrated with the approved NGEN acquisition strategy. Program officials  told us they did not revise the dates in the master schedule until after the  continuity of services contract was awarded (July 2010), and that the dates  in the acquisition strategy reflected the current information. By using a  source other than the program office’s working schedule, oversight  officials’ expectations about when milestones will be met may not be  realistic.
Establishing the critical path for all activities. None of the four  subschedules fully met this practice. Specifically, the scheduling tool was  unable to generate a valid critical path for the subschedules due to the  extent of issues associated with the sequencing of activities, integration of  activities, and identification of reasonable float (discussed below).  Program officials stated that they do not manage a critical path generated  by the scheduling tool. Instead, these officials stated that they track  activities associated with the deployment phase decision (Milestone C),  which they have designated as being critically important to them.  However, such practice does not allow the program to have immediate  insight into the full sequence of activities (both critical and noncritical)  that, if delayed, would impact the planned completion date of Milestone C,  as well as a projected completion date should one of these activities be  delayed.
Identifying reasonable float between activities. Two subschedules  partially met this practice, while the remaining two minimally met it. Each  of these subschedules identified float; however, the amount of excessive  float varied. Both the Contract Technical Representative Workforce  Reconstitution and IT Service Management Process Development  subschedules partially met this practice because only 25 percent and 41  percent of their work activities had float of 100 days or greater,  respectively. The two remaining subschedules minimally met this practice  because over 60 percent of their activities contained float of 100 days or  greater.
Excessive float values are indicative of schedule logic that is flawed,  broken, or absent. As such, these float values are of limited value to  mitigate risk by reallocating resources from tasks that can safely slip to  tasks that must be completed on time.
Conducting a schedule risk analysis. The program has not performed a  schedule risk analysis. Instead, according to program officials, schedule  risks are considered during risk management board meetings and program  health assessments. However, without this analysis, it is not possible to  determine a level of confidence in meeting program milestones. A  schedule risk analysis will calculate schedule reserve, which can be set  aside for those activities identified as high-risk. Without this reserve, the  program faces the risk of delays if they were to occur on critical path  activities.
Updating the schedule using logic and durations. All four subschedules  partially met this practice. According to program officials, status updates  are performed on the subschedules once a week. However, despite status  updates, date anomalies exist. For example, the Contract Technical  Representative Workforce Reconstitution subschedule included five  activities with an actual start date in the future. Furthermore, the  subschedules’ inability to produce a valid critical path indicates that the  sequencing of activities is not appropriate, thus impairing the scheduling  tool’s ability to generate realistic start and end dates.
According to program officials, they were aware of some of these schedule  weaknesses based on a May 2010 assessment of the schedule performed  by a support contractor. Among other things, the contractor’s assessment  found that the schedule did not provide for stakeholder review of most of  the major acquisition documents or steps to mitigate known risks, and that  it lacked a valid critical path due to network logic issues and activity  constraints. Officials told us that they plan to address these issues.
In addition, program officials stated that they hold monthly program  management reviews to discuss schedule quality issues, as well as risks or  issues that might affect the schedule. However, these reviews are not  addressing key schedule issues. Specifically, the NGEN schedule  management plan calls for the schedule to be resource-loaded from a  centralized resource pool approved by the program manager, activities  beginning within 90 days to have durations of no more than 20 days, and  activities for mitigating approved program risks to be added to the  schedule. However, our analysis of the schedule showed that  resources are not assigned within the schedule,   activities that are to begin within 90 days have durations that exceed 20    activities for mitigating 10 approved program risks were not included.
Collectively, the weaknesses in implementing the nine key practices for  the program’s integrated master schedule increase the risk of schedule  slippages and related cost overruns and make meaningful measurement  and oversight of program status and progress, as well as accountability for  results, difficult to achieve. Moreover, they undermine the schedule’s  ability to produce credible dates for planned NGEN milestones and events.  In the case of increment 1, this risk has already been realized. Specifically,  the NGEN master schedule was rebaselined in August 2010, resulting in  delays in a number of key dates, including a 5-month delay of the  Milestone C decision. See table 12 for a summary of key event and  milestone delays.
While officials stated that they have addressed some of the weaknesses  identified above in the August 2010 rebaselined integrated master  schedule, they conceded that this schedule does not assign resources to  work activities, and the scheduling tool is unable to generate a valid  critical path. Because these key scheduling practices are not being  performed, the schedule is still not reliable.
Without a fully integrated and reliably derived schedule for the entire  NGEN program, the program office cannot identify when and how it will  proceed through Milestone C and ultimately transition from NMCI to  NGEN, and it cannot adequately manage and measure its progress in  executing the work needed to do so.
section_title: NGEN Acquisition Decisions Were Not Always Performance- and Risk-Based
paragraphs: Successful execution of system acquisition programs depends in part on  effective executive-level governance, to include having organizational  executives review these programs at key milestones in their life cycles and  make informed performance- and risk-based decisions as to how they  should proceed. DON policy recognizes the importance of such milestone  reviews. According to this policy, acquisition programs must proceed  through a series of gate reviews (as discussed above), during which  program performance is assessed and satisfactory program health must be  demonstrated prior to moving forward.
Currently, program performance and health at each gate are assessed  using the Naval Probability of Program Success assessment methodology,  which was established in September 2008. This assessment addresses four  aspects of a program: (1) requirements, (2) resources, (3)  planning/execution, and (4) external influencers. Associated with each  aspect are two or more metrics, each of which is scored based on  underlying criteria that are unique to each gate. (See table 13 for a  description of each metric.) At a given gate review, the criteria are rated in  terms of green, yellow, or red. Further, the metrics can be designated as  critical, meaning that any significant issues that are associated with these  metrics must be resolved before the gate can be exited.
As noted earlier, a Gate 1 review was not held because the gate-review  process was not established when the program began. In lieu of a Gate 1  review, according to the NGEN Acquisition Strategy, the Chief of Naval  Operations Executive Board met to confirm NGEN requirements during  the winter of 2007/2008 and these meetings were “nominally” a Gate 1  review. Subsequent to the establishment of the DON gate process, an  NGEN Gate 2 review—intended to focus on an analysis of alternatives— was waived in early 2008 because the department planned to continue the  use of existing NMCI technology, and NGEN entered the DON review  process at Gate 3 in April 2008. OSD later identified the program as a pre- MAIS acquisition, resulting in the direction to conduct an analysis of  acquisition alternative approaches. As such, DON held a Gate 2 review in  April 2009, one year after the Gate 3 review. Since then, DON held a Gate 4  review in November 2009, as well as a Gate 5 review in October 2010. As  discussed below, the extent to which each of the gate reviews was  performance- and risk-based varied.
Gate 3 review. At the time of this review, which was in April 2008, the  Probability of Program Success assessment methodology was not yet in  place. Therefore, program review documentation focused on, for example,  program activities that had been completed, were under way, and were  planned. However, these activities were not presented relative to any  benchmark or goal, and thus program performance was not apparent in  the documentation. Further, while program documentation shows that  risks were disclosed, such as funding shortfalls for fiscal years 2008 and  2009, as well as workforce and training challenges, the scope and nature of  the information presented did not extend to the level that the assessment  methodology provides. For example, the information presented did not  address the realism and achievability of the program master schedule and  the confidence level associated with the current cost estimate, including  the difference between the program office and independent cost estimates,  which are both relevant criteria under the assessment methodology for the  gate. Notwithstanding these gaps in information that would have limited  informed program decision making, the program was approved to  proceed.
Gate 2 review. At the time of this review, which was in April 2009, the  Probability of Program Success assessment methodology was in place.  However, it was not used to inform program decision making. Instead, the  review focused on the AOA, next steps, and the overall program timeline.  While briefing documentation shows that cost estimates for the  alternatives exceeded planned funding, the documentation did not  disclose the range of AOA and integrated master schedule weaknesses  discussed earlier in this report, and the risks associated with these  limitations. This is significant because the Gate 2 assessment criteria focus  on, among other things, whether the AOA cost estimates and master  program schedule are reliable and whether program execution is on  schedule. Notwithstanding these weaknesses, the program was approved  to proceed.
Gate 4 review. For this review, DON used its Probability of Program  Success methodology and assessed the health of the program against each  of the 17 metrics, including 3 that DON designated as potentially  critical—parameter status, budget and planning, and acquisition  management. According to the program health assessment used at this  gate, 8 of the 17 metrics were rated as red, meaning that the program had  identified significant issues that would inhibit delivery of capability within  approved cost and schedule constraints and that mitigation strategies had  not been identified. Moreover, the 8 metrics rated as red included 3 that  were designated as critical, meaning that these issues needed to be  resolved before exiting the gate. Specifically, the parameter status metric  was rated as red because NGEN requirements that increment 1 is to meet  had not yet been defined; the budget and planning metric was rated as red  because the program was not fully funded; and the acquisition  management metric was rated as red because the USD (AT&L) had yet to  authorize the milestone at which the program would enter the Defense  Acquisition System. (See fig. 4 for the assessment results for all 17  metrics.) Moreover, the gate briefing document highlighted a number of  risks facing the program. For example, it faced the risk that key program  documentation, such as the System Engineering Plan and the Test and  Evaluation Master Plan, would not be completed until NGEN requirements  were defined. Further, it faced the risk that insufficient funding would  impact the program office’s ability to acquire NMCI assets. Nevertheless,  the program was approved to proceed.
Gate 5 review. For this review, which was conducted in October 2010,  DON again used its Probability of Program Success methodology and  assessed program performance and risk against all 18 metrics, including 9  that DON designated as potentially critical. Three metrics were rated as  red; 1, test and evaluation, was deemed critical. According to the  assessment, the test and evaluation metric was rated as red because the  Test and Evaluation Master Plan was not complete; the budget and  planning metric was rated as red because of significant NGEN funding  reductions; and the manning metric was rated as red because of  inadequate program office contracting, engineering and logistics  personnel. Further, according to the assessment, the Test and Evaluation  Master Plan was not complete because the requirements were not defined.  As discussed above, the program recognized, at Gate 4, the risk that a  delay in defining NGEN requirements would impact the completion of this  plan. (See fig. 5 for the assessment results for all 18 metrics.) According to  the gate briefing document, these red ratings introduced a number of risks,  such as the risk that the program would not be able to execute its current  acquisition approach and meet program milestones.
In addition, even though the assessment rated the acquisition management  metric as green, this rating is not consistent with our findings in this report  about the NGEN integrated master schedule. Specifically, the rationale for  the green rating was that the August 2010 rebaselined schedule was  viewed as realistic and achievable by key stakeholders. However, as stated  earlier, program officials conceded that the schedule does not assign  resources, and the scheduling tool is unable to generate a valid critical  path, which are key scheduling practices; thus the August 2010 schedule  was not reliable. The approval of the Assistant Secretary of the Navy  (Research, Development and Acquisition) for NGEN to proceed beyond  Gate 5 was made conditional on the program satisfactorily completing  action items focused on releasing the request for proposals for the  Transport Services contract (scheduled for January 2011) and resolving its  funding shortfall.
As shown above, DON has demonstrated a pattern of approving NGEN at  key acquisition review gates in the face of both limited disclosure of the  program’s health and risks and known program risks and shortfalls in  performance. According to DON officials, the decisions to pass the gates  and proceed were based on their view that they had sufficiently mitigated  known risks and issues. By not fully ensuring that NGEN gate decisions  sufficiently reflected program challenges, DON has increased the  likelihood that the NGEN acquisition alternative that it is pursuing is not  the most cost-effective course of action, and that the program will cost  more and take longer to complete than planned.
section_title: Conclusions
paragraphs: Given the enormous size, complexity, and mission importance of NGEN, it  is vital that DON and DOD assure decision makers, including the  congressional defense committees, that the approach to acquiring needed  capabilities is the most cost-effective and that its execution is guided by a  well-defined schedule and informed milestone decision making. To date,  this has not occurred to the degree that it should. Most notably, while  DON produced substantially well-documented cost estimates, the NGEN  acquisition approach currently being followed is not grounded in a reliable  analysis of alternative approaches, and the selected approach was not  even assessed and is about $4.7 billion costlier and introduces more risk  than the alternatives that were assessed. Further, the program’s execution  to date has not been based on the kind of reliably derived integrated  master schedule that is essential to program success. While the program  office is aware of some of the schedule weaknesses and intends to address  them, additional work is needed to ensure that the schedule can produce  credible dates for planned NGEN milestones and events. Exacerbating this  is an equally troubling pattern of missed milestones and delays in key  program documentation, as well as gate review decisions that have  allowed the program to proceed in the face of significant performance  shortfalls and risks.
While NGEN is scheduled for an OSD-level milestone review in August  2011, the above schedule limitations make it likely that this review date  will slip. It is thus imperative, given the scope and nature of the program’s  problems, that swift and immediate action be taken to ensure that the  most cost-effective acquisition approach is pursued and that a reliable  schedule and performance- and risk-based decision making are employed.  To do less increases the chances that needed NGEN capabilities will be  delivered late and be more costly than necessary.
section_title: Recommendations for Executive Action
paragraphs: To ensure that NGEN capabilities are acquired in the most cost-effective  manner, we recommend that the Secretary of Defense take the following  two actions:    direct the Under Secretary of Defense for Acquisition, Technology, and  Logistics to conduct an interim NGEN milestone review, and    direct the Secretary of the Navy to immediately limit further investment in  NGEN until this review has been conducted and a decision on how best to  proceed has been reported to the Secretary of Defense and congressional  defense committees.
At a minimum, this review should ensure that DON pursues the most  advantageous acquisition approach, as evidenced by a meaningful analysis  of all viable alternative acquisition approaches, to include for each  alternative reliably derived cost estimates and metrics-based operational  effectiveness analyses. In addition, the review should consider existing  performance shortfalls and known risks, including those discussed in this  report.
To facilitate implementation of the acquisition approach resulting from the  above review, we further recommend that the Secretary of Defense direct  the Secretary of the Navy to take the following two actions:    ensure that the NGEN integrated master schedule substantially reflects the  key schedule estimating practices discussed in this report, and    ensure that future NGEN gate reviews and decisions fully reflect the state  of the program’s performance and its exposure to risks.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Deputy  Assistant Secretary of Defense (C3, Space and Spectrum), and reprinted in  appendix II, DOD stated that it concurred with one of our four  recommendations, did not concur with one recommendation, and partially  concurred with two. The department’s comments are discussed below.
The department partially concurred with our recommendation to conduct  an interim milestone review that provides assurance that DON is pursuing  the most advantageous acquisition approach. Specifically, the department  stated that it intended to leverage the next OSD-chaired NGEN  Overarching Integrated Product Team meeting in February 2011 for the  review and that following this meeting, the USD(AT&L) will conduct a  Milestone Decision Authority review of the current NGEN approach, along  with risks. According to the department, this approach balances the  review processes already in place, resource constraints, and the need for  an additional milestone review. Further, the department said it had  concluded that DON’s AOA was sufficient and that the analysis had been  approved by CAPE. DOD added that it will complete an economic  analysis—a post AOA-activity—for the August 2011 Milestone C review,  which will include a follow-on independent cost estimate and an updated  determination of the most cost-effective solution.
While these are important steps, DOD’s planned actions do not appear to  fully address our recommendation. Specifically, the department did not  indicate any intent to reevaluate whether the current solution is indeed the  most advantageous approach, despite the weaknesses contained in the  AOA identified in this report and the fact that the current approach was  not included in its analysis. According to the September 2010 draft NGEN  economic analysis development plan, only the status quo and the current  approach are to be analyzed, not the other three alternatives that were  included in the AOA. Without a meaningful analysis of alternatives, DOD  will be unable to determine the most cost-effective solution in its two  upcoming key reviews.
The department did not concur with our recommendation that it limit  further investment in NGEN until a decision has been made on how best to  proceed based on an interim review that considers all viable alternative  acquisition approaches and this decision has been reported to the  Secretary of Defense and to congressional defense committees. The  department stated that DON’s NGEN acquisition strategy and program  management have been approved by the milestone decision authority, and  that adequate oversight is in place to ensure regulatory and statutory  compliance. Further, the department said that the limitation on NGEN  investments will impact future DON business operations and, ultimately,  Naval warfighting capabilities. The department added that it will make  adjustments to NGEN investments if it determines they are required;  however, it also said it must continue to execute the investments within  the time frame of the continuity of services contract.
While oversight is in place for the NGEN program, it is not effective.  Specifically, as discussed in this report, DON’s past reviews have resulted  in decisions that were not always performance- and risk-based. Given that  DON is continuing to proceed in the face of the problems we are reporting,  it is even more important that adequate oversight be provided by the  Secretary and congressional defense committees.
Moreover, we maintain that limiting further investment in NGEN, thereby  delaying the Milestone C event and its associated activities, is the most  prudent action at this time. By not evaluating all viable acquisition  approaches before proceeding with further investment in NGEN, the  department cannot be assured that it is pursuing the most cost-effective  approach. Further, by selecting an approach that, as discussed in this  report, carries greater relative schedule and performance risks than other  alternatives and is being executed against an unreliable program schedule,  the department increases the risk that its approach will lead to future cost  overruns, requiring it to expend additional resources that could otherwise  be used to provide other warfighting capabilities. Furthermore, even if the  department proceeds along its current course, the issues we have  identified with the program’s schedule, along with the delays already  experienced, raise concerns that it will be unable to complete the  transition as planned within the time frames of the current continuity of  services contract.
The department partially concurred with our recommendation that the  Secretary of Defense direct the Secretary of the Navy to ensure that the  NGEN integrated master schedule substantially reflects the key schedule  estimating practices discussed in this report. DOD stated that the  integrated master schedule was developed in accordance with industry  best practices. However, as discussed in this report, none of the  subschedules that we analyzed reflected all the practices that our work  has identified as necessary to develop and maintain a reliable schedule. To  its credit, DOD also said it would seek ways to improve schedule  performance and that DON will review the scheduling practices discussed  in this report and incorporate those found to be beneficial. We continue to  believe that the Secretary of the Navy should ensure that the NGEN  integrated master schedule incorporates all of the best practices for  schedule estimating discussed in this report to help manage and measure  its progress in executing the work needed to proceed through Milestone C  and ultimately transition from NMCI to NGEN.
The department concurred with our recommendation to ensure that future  NGEN gate reviews and decisions fully reflect the state of the program’s  performance and its exposure to risks. In this regard, the department  stated that it plans to continue to conduct monthly risk management board  meetings and program health reviews, and report the results to program  leadership. It will be critical that decisions on NGEN fully reflect the state  of the program’s performance and exposure to risks.
We are sending copies of this report to the appropriate congressional  committees; the Director, Office of Management and Budget; the  Congressional Budget Office; the Secretary of Defense; and the Secretary  of the Navy. The report also is available at no charge on the GAO Web site  at http://www.gao.gov.
If you or your staff members have any questions on matters discussed in  this report, please contact me at (202) 512-6304 or melvinv@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine whether (1) the Department of the Navy  (DON) sufficiently analyzed alternative approaches for acquiring its Next  Generation Enterprise Network (NGEN), (2) DON has a reliable program  schedule for executing NGEN, and (3) acquisition decisions have been  performance- and risk-based.
To address the first objective, we evaluated the analysis of alternatives  (AOA) report and its supporting documentation against relevant  Department of Defense (DOD) guidance and GAO’s Cost Estimating and  Assessment Guide and compared the alternatives in the AOA final report  with the NGEN Acquisition Strategy. More specifically,    For the cost analysis, we compared the AOA cost estimating  documentation, such as the cost model spreadsheet, supporting  documentation for the cost model, and the final NGEN AOA report,  against the four characteristics of a reliable estimate in GAO’s Cost  Estimating and Assessment Guide to determine the extent to which the  cost estimates reflected each of the four characteristics.
For the operational effectiveness analysis, we compared an NGEN  alternatives performance assessment report and the AOA final report  against the relevant DOD guidance to determine the extent to which the  analysis was sufficient. In addition, we reviewed NGEN AOA Advisory  Group meeting minutes and documentation containing the results of a  Space and Naval Warfare Systems Command review of the cost analysis.  We also interviewed cognizant DON and Office of the Secretary of Defense  officials about the AOA’s development and results.
To address the second objective, we first reviewed the integrated master  schedule and 4 of the 29 subschedules that existed when we began our  review and that comprised the early transition activities intended to  address key program risks, as well as high-level plans for  postdeployment. Accordingly, we focused on assessing the May 2010  subschedules against the nine key schedule estimating practices in GAO’s Cost Estimating and Assessment Guide using commercially available  software tools to determine the extent to which each subschedule  reflected each of the nine practices (e.g., a logical sequence of activities  and reasonable activity durations). Further, we characterized the extent  which each subschedule satisfied each of the practices as either met,  substantially met, partially met, minimally met, or not met. In addition, compared the baseline schedule, established in December 2009, to the  rebaselined schedule, established in August 2010, to identify whether key  event and milestone dates had slipped. We also interviewed cognizant  officials about development and management of the integrated master  schedule and underlying subschedules. We also reviewed program  documentation, such as the NGEN schedule management plan, program  performance reports, program management reviews, and the acquisition  strategy.
To address the third objective we compared program review  documentation, such as briefings, program performance assessments, and  meeting minutes, to DON acquisition review policies and procedures, as  well as to other programmatic documents, such as risk registers and risk  management board briefings and meeting minutes. We also interviewed  cognizant program officials regarding NGEN performance and program  risks.
To assess the reliability of the data that we used to support the findings in  this report, we reviewed relevant program documentation to substantiate  evidence obtained through interviews with agency officials. We  determined that the data used in this report are sufficiently reliable. We  have also made appropriate attribution indicating the sources of the data.
We conducted this performance audit at DOD offices in the Washington,  D.C., metropolitan area and at the Space and Naval Warfare Systems  Command in San Diego, California, from October 2009 to February 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  were Randolph C. Hite, Director; Carol Cha, Assistant Director; Monica  Anatalio; Mathew Bader; Neil Doherty; Cheryl Dottermusch; James Houtz;  Kaelin Kuhn; Neela Lakhmani; Lee McCracken; Jeanne Sung; and Adam  Vodraska.