DOD Major Automated Information Systems: Improvements Can Be Made in Reporting Critical Changes and Clarifying Leadership Responsibility
Released: 2016-03-30T00:00:00

Letter

Background

DOD’s organizational structure includes the Office of the Secretary of Defense, the Joint Chiefs of Staff, the military departments, numerous defense agencies and field activities, and various unified combatant commands that contribute to the oversight of DOD’s acquisition programs. Figure 1 provides a simplified depiction of DOD’s organizational structure.

The Under Secretary of Defense for AT&L serves as the Defense Acquisition Executive and the Under Secretary has responsibility for oversight of MAIS acquisition programs. AT&L has policy and procedural authority for the defense acquisition system, which establishes the steps that DOD programs generally take as DOD plans, acquires, deploys, operates, and maintains its IT systems (discussed in more detail following this section). Additionally, AT&L is the principal acquisition official of the department and is the acquisition advisor to the Secretary of Defense. AT&L’s authority includes directing the military services and defense agencies on acquisition matters and making milestone decisions for MAIS programs. AT&L can delegate decision authority for MAIS programs to a component head who may further delegate the authority to the component acquisition executive.

DOD’s CIO is the Principal Staff Assistant and senior IT advisor to the Secretary of Defense. This role includes overseeing many national security and defense business systems and managing information resources. The CIO coordinates with AT&L to develop and maintain a process for assessing and managing the risks related to the department’s IT acquisitions, including MAIS programs.

DOD’s Acquisition Guidance and Framework for Managing MAIS Programs

The Department of Defense Instruction 5000.02 establishes policy for the management of all DOD acquisition programs. In January 2015, DOD updated these guidelines which outline the framework for MAIS programs. This framework consists of six models for acquiring and deploying a program, including two hybrid models that each describe how a program may be structured based on the type of product being acquired (e.g., software-intensive programs and hardware-intensive programs). A generic acquisition model that shows all of the program life-cycle phases and key decision points is shown in figure 2 and described following the figure.

Materiel solution analysis: Refine the initial system solution (concept) and create a strategy for acquiring the solution. A decision—referred to as milestone A—is made at the end of this phase to authorize entry into the technology maturation and risk reduction phase.

Technology maturation and risk: Determine the preferred technology solution and validate that it is affordable, satisfies program requirements, and has acceptable technical risk. A decision—referred to as milestone B—is made at the end of this phase to authorize entry of the program into the engineering and manufacturing development phase and award development contracts. An acquisition program baseline is first established at the milestone B decision point or at program initiation, whichever occurs later. A program’s first acquisition program baseline contains the original life-cycle cost estimate (which includes acquisition and operations and maintenance costs), the schedule estimate (which consists of major milestones and decision points), and performance parameters that were approved for that program by the milestone decision authority. The first acquisition program baseline is established after the program has refined user requirements and identified the most appropriate technology solution that demonstrates that it can meet users’ needs.

Engineering and manufacturing development: Develop a system and demonstrate through testing that the system meets all program requirements. A decision—referred to as milestone C—is made during this phase to authorize entry of the system into the production and deployment phase or into limited deployment in support of operational testing.

Production and deployment: Achieve an operational capability that meets program requirements, as verified through independent operational tests and evaluation, and implement the system at all applicable locations.

Operations and support: Operationally sustain the system in the most cost-effective manner over its life cycle.

Statutory Reporting Requirements for MAIS Programs

MAIS programs enable DOD to organize, plan, direct and monitor important mission operations. As previously mentioned, MAIS programs must comply with certain annual and quarterly reporting requirements identified in statute. Each calendar year, DOD must submit to Congress budget justification documents on each MAIS program, including information on cost, schedule, and performance. Specifically, these programs must report, among other things, on the development and implementation schedules and total acquisition and full life-cycle cost estimates and provide a summary of the key performance parameters for each program. DOD must also provide a summary of any significant changes to information previously provided for each program.

Moreover, on a quarterly basis, the program manager for each MAIS program is required to provide the senior DOD official responsible for the program a written report that identifies any variance in the program’s cost, schedule, or performance. Depending on the determination after reviewing the variances identified in the quarterly report, the senior DOD official responsible for the program must notify the congressional defense committees of any programs that have experienced either a significant or critical change. During our review, MAIS programs were required to comply with the following reporting requirements:

Significant change. A significant change must be declared if a program experienced a schedule delay of more than 6 months but less than a year; estimated total acquisition or full life-cycle cost for the program has increased by at least 15 percent but less than 25 percent; or there has been a significant adverse change in the expected performance of the system. If such an event occurs, the senior DOD official responsible for the program must notify the congressional defense committees in writing no later than 45 days after receiving the quarterly report from the program manager.

Critical change. A critical change must be declared if a program failed to achieve a full deployment decision within 5 years after the milestone A decision or, if there was no milestone A decision, the date when the preferred alternative was selected for the program; experienced a schedule delay of 1 year or more; experienced an estimated total acquisition or full life-cycle cost increase of 25 percent or more over the original estimate; or experienced a change in the expected performance of the system that will undermine the ability of the system to perform as intended. If such an event occurs, the senior DOD official responsible for the program must carry out an evaluation and submit a critical change report to the congressional defense committees no later than 60 days after receiving the quarterly report.

Since the December 19, 2014, enactment of the Carl Levin and Howard P. “Buck” McKeon National Defense Authorization Act for Fiscal Year 2015, MAIS programs are now required to declare a significant change— instead of a critical change—if they fail to achieve a full deployment decision within 5 years after the milestone A decision, the date when the preferred alternative was selected for the program (excluding any time during which program activity is delayed as a result of a bid protest).

More recently, the National Defense Authorization Act for Fiscal Year 2016 directed the Secretary of Defense to issue guidance for MAIS programs to establish an acquisition baseline within 2 years after program initiation. This statute provides a response to a recommendation we made in our last annual report on MAIS programs. In particular, we found that these programs spent, on average, more than 5 years and $450 million prior to establishing baselines. We noted that programs that have not established baselines were subject to less oversight and could not be measured against cost, schedule, and performance targets. Also, the propensity to carry out MAIS programs for multiple years prior to committing to baselines is inconsistent with incremental and rapid development as called for in federal law and GAO’s IT management best practices. Accordingly, we recommended that these programs be baselined within 2 years; for which DOD partially concurred. We maintained that establishing baselines within 2 years would improve outcomes and increase accountability.

OMB’s IT Dashboard Codified to Provide Transparency into the Performance of Agency IT Investments

DOD’s CIO, along with other agencies, must report on the progress of its IT investments, including MAIS programs, on a public website known as the IT Dashboard. OMB established this website in June 2009 to improve the transparency and oversight of agencies’ investments. The Dashboard visually displays federal agencies’ cost, schedule, and performance data for over 700 major federal investments at 26 federal agencies. It also includes a risk rating that is to be performed by agency CIOs. According to OMB, these data are intended to provide a near-real- time perspective on the performance of these investments.

The public display of agency data is intended to allow OMB; other oversight bodies, including Congress; and the general public to hold federal agencies accountable for their progress and results. In August 2011, OMB issued guidance that stated, among other things, that agency CIO’s shall be held accountable for the performance of IT investments. The Dashboard presents performance ratings for individual investments using metrics that OMB has defined—cost, schedule, and CIO evaluation. If OMB or the agency CIO determine the reported data is not timely or reliable, the CIO must notify OMB and establish within 30 days of this determination an improvement program and the progress the agency is making. According to OMB, the addition of CIO names and photos on the website is intended to highlight this accountability and link the Dashboard’s reporting on investment performance.

In order to enhance transparency and improve risk management of federal IT acquisitions, Congress codified the Dashboard reporting process through key provisions, known as the Federal Information Technology Acquisition Reform provisions in the Carl Levin and Howard P. “Buck” McKeon National Defense Authorization Act for Fiscal Year 2015.

Best Practices for Managing IT Acquisition Programs

Entities such as the Project Management Institute, the Software Engineering Institute at Carnegie Mellon University, and GAO have developed and identified best practices to help guide organizations to effectively plan and manage their acquisitions of major IT systems, such as MAIS programs. Our prior reviews have shown that properly applying such practices can significantly increase the likelihood of delivering promised system capabilities on time and within budget. These practices include, but are not limited to:

Requirements management: Requirements establish what the system is to do, how well it is to do it, and how it is to interact with other systems. Appropriate requirements development involves eliciting and developing customer and stakeholder requirements, and analyzing them to ensure that they will meet users’ needs and expectations. It also consists of validating requirements as the system is being developed to ensure that the final systems to be deployed will perform as intended in an operational environment.

Risk management: A process for anticipating problems and taking appropriate steps to mitigate risks and minimize their impact on program commitments. It involves identifying and documenting risks, categorizing them based on their estimated impact, prioritizing them, developing risk mitigation strategies, and tracking progress in executing the strategies.

All Programs with a Critical Change Reported Required Elements, but Most Missed Deadlines and Obligations Were Not Monitored

According to statute, for programs that declare a critical change, the report that is submitted to Congress must include a written certification stating that: the automated information system or IT investment to be acquired is essential to the national security or to the efficient management of the DOD; there is no alternative to the system or IT investment which will provide equal or greater capability at less cost; the new estimates of the costs, schedule, and performance parameters have been determined, with the concurrence of the Director of Cost Assessment and Program Evaluation, to be reasonable; and the management structure for the program is adequate to manage and control program costs.

All 18 MAIS critical change reports in our review contained the required elements.

In addition, this report must be prepared and submitted to Congress no later than 60 days after the senior DOD official responsible for the program receives the quarterly report from the program manager that leads to the determination that a critical change event has occurred. Programs that do not submit the report to Congress within the 60-day period are statutorily prohibited from obligating appropriated funds for any major contract until the date that Congress receives the report. Further, if a MAIS program violates the statutory prohibition against obligations, it will also violate the Antideficiency Act. This act prohibits an officer or employee of the United States government from making or authorizing an expenditure or obligation in excess of or in advance of available appropriations. The Antideficiency Act also requires that an appropriation must be available for an agency to incur an obligation. Thus, if DOD incurs an obligation against an appropriation that is not legally available, the department has violated the act. Violating the Antideficiency Act would require the Secretary of Defense to immediately report to the President and Congress all relevant facts and a statement of actions taken.

Of the 18 MAIS programs experiencing a critical change, most exceeded the 60-day reporting requirement, several by a substantial amount. Specifically, 16 exceeded the 60-day reporting requirement and 10 of those programs took over 100 days to report. Of the 10 programs, 5 of the programs took over 200 days to report. Two programs—Teleport Gen I/II and General Fund Enterprise Business Systems—delivered their reports to Congress within the 60-day requirement. Figure 3 shows the extent that programs met or exceeded the 60-day reporting requirement.

Officials from several programs provided various reasons for why they delayed submitting their critical change reports. For example:

Mission Planning Systems #2 submitted a report to the Office of the Secretary of Defense on time but it took 29 days to transmit the package to Congress and it was 3 days late.

CAC2S program was late because of the need to conduct an independent assessment that was directed by DOD.

The Expeditionary Combat Support System program had delays due to changes that were made to the size and complexity of its originally scoped effort and contracting process that, in turn, required additional updates. The update triggered a process to re-evaluate the revised strategy.

According to a DOD AT&L official, 60 days is too short to perform a program evaluation and achieve all the coordination necessary for an important communication with Congress. The official also said the addition of the Office of Cost Assessment and Program Evaluation requirement to review and approve the reports, as mandated by the Weapon Systems Acquisition Reform Act of 2009, consumes much of the 60-day allotment time. A Cost Assessment and Program Evaluation official noted that reviews often exceed the 60-day period because, most notably, the significant amount of time needed to collect and develop comprehensive information used to determine a program’s cost. The official added that DOD is working to strengthen the data collection efforts to improve the ability of the Office of Cost Assessment and Program Evaluation to complete its evaluation such as reviewing the basis for the revised cost and schedule estimates. However, the official noted that there has been no overall evaluation or study on the cause for delays.

While it may be possible that 60 days is too short of a time frame for submitting reports, without understanding the cause for the delays DOD is not in a position to state what time frame would be feasible. Further, the fact that two programs were able to submit reports in a timely manner suggests that 60 days is achievable. Until DOD ascertains the cause for the delays and implements corrective actions, the reports may continue to be delivered in a manner which may impact the timeliness of information considered by Congress in making oversight and funding decisions for MAIS programs. This may also affect budget and other strategic decisions on how and what programs to prioritize.

Further, DOD does not have a mechanism to monitor and ensure that MAIS programs with late reports were restricted as required by law from obligating funds on major contracts prior to Congress receiving the report. This mechanism is especially important because of the potential for violating the Antideficiency Act. Although DOD states in its guidance that program managers should not obligate any funds during the entire period in which the report is being prepared, DOD does not currently have a way to monitor this.

According to a DOD AT&L official, DOD is not required by statute, regulation, or guidance to collect the information for monitoring purposes. However, our guidance on internal controls for federal agencies states that agency management should establish a baseline to monitor the current state of a control system. Once established, management should monitor the agencies’ internal control system through ongoing monitoring and separate evaluations DOD does not have a management internal control to monitor the system and evaluate whether programs are complying with the DOD guidance. Instead, DOD relies on the programs to act in accordance with the law. This official said the need to obligate funds on major contracts should be a driver for programs to submit their reports to Congress as expeditiously as possible. However, with so many programs submitting the critical change reports well after the 60-day period, there is a risk that programs could potentially violate the prohibition on obligations, and thus, in addition, the Antideficiency Act.

Selected MAIS Programs Had Varying Cost and Schedule Estimate Changes, One Did Not Meet Its Technical Performance Targets

The extent to which the three selected MAIS programs in our study experienced changes in their cost and schedule estimates and met performance targets varied. Specifically, the Army and Air Force programs experienced slight changes in their cost and schedule estimates, while the Navy program experienced more significant changes. In addition, only one program, Air Force’s DEAMS, did not fully meet its technical performance targets. Table 1 provides a status of the cost, schedule changes, and the results of technical performance targets for the programs. See appendix II for the detailed profiles of each program.

Army’s Program Had a Slight Cost Increase, but Was Within Schedule and Met All Its Performance Targets

As of January 2016, the latest life-cycle cost estimate for the Army’s TMC program had increased about 19 percent from the program’s February 2008 acquisition program baseline estimate (from approximately $1.97 billion up to $2.34 billion). Program officials attributed the cost increase to a breach in the research and development testing and evaluation cost estimation that was reported to Congress. The program’s estimated program development cost increased by 45 percent over the original acquisition program baseline due to program scope changes derived from the realignment of certain missions, such as the endorsement of the Command Post of the Future as a foundation for mission command.

As of January 2016, TMC program experienced a 3-month slippage in its full deployment date, currently scheduled for December 2018. The slippage was within the program’s pre-established threshold allowance to account for minor changes in schedule and, program officials stated that this slippage was considered to be a low risk that the program has accepted. Program officials stated that, although the Command Post of the Future product is 95 percent fielded and is on schedule to reach full deployment by December 2018, continued support of the Command Post Computing Environment is needed beyond fiscal year 2019.

As of January 2016, the TMC program met all three of its key performance targets, which include supporting net-centric military operations, disseminating orders with future Army and Joint Command and Control Systems, and displaying unified information on subject matters.

The Navy Program Experienced Significant Cost and Schedule Increases, but Met All Performance Targets

As of October 2015, the latest life-cycle cost estimate for Navy’s CAC2S Increment 1 program had increased about 477 percent from its first acquisition program baseline estimate (from approximately $347 million up to $2 billion). As previously reported, factors attributed to the program’s early developmental challenges contributed to an increase in its cost estimate, which include program scope growth and restructuring. According to program documentation, despite the program’s initial challenges in its cost estimates due to an increase in its operations and support expenditures, the CAC2S program has demonstrated gradual improvement as it reported a cost avoidance of $54.4 million for implementing the DOD Better Buying Power initiatives that befitted from competitive market forces that drove down cost. As of October 2015, the program’s latest life-cycle estimate relative to its November 2010 production acquisition program baseline cost estimate had decreased by about 19 percent (from approximately $2.46 billion down to $2 billion).

As of October 2015, compared to its first acquisition program baseline schedule, the program experienced a 13 year and 9 month slippage in its full deployment date—currently scheduled for March 2022. As previously reported, factors that attributed to the prior schedule slippage included the addition of new requirements and program restructuring. However, program officials stated that the program has been executing in accordance within its approved schedule. As of October 2015, the estimated milestone C phase 2 was delayed by 6 months from the program’s production acquisition program baseline but achieved it milestone approved within the program’s pre-established schedule threshold. Program officials attributed this delay, in part, to administrative factors, which included the review and approval process. CAC2S successfully achieved milestone C phase 2 approval in February 2015 but the acquisition decision memorandum had not been signed until March 2015.

As of October 2015, program officials reported that, during performance testing, it was meeting both of its key performance targets related to net- ready and data fusion.

Air Force’s Program Had Slight Cost and Schedule Increases and Did Not Meet Performance Targets

As of October 2015, the latest life-cycle cost estimate for the Air Force’s DEAMS program had increased about 9 percent from its first February 2012 acquisition program baseline estimate (from approximately $1.43 billion up to $1.56 billion). Program officials attributed the cost increase, in part, to program scope growth and the addition of software upgrade enhancements. Specifically, as of October 2015, the program’s life-cycle cost estimate incorporated additional infrastructure maintenance costs throughout the life-cycle that added performance monitoring and additional deployment support. Also, according to program officials, the program brought forward increment 2 requirements and a second Oracle software upgrade in year 2021.

DEAMS experienced a 6 month slip in its milestone C but was within its threshold, a predefined point where programs that exceed it are at increased risk. However, it did experience a 1 year slip in its full deployment decision date—currently scheduled for February 2016. Program officials attributed this slippage due to findings identified DEAMS’s initial operational test and evaluation report. While the program had been established since August 2003, a full deployment date had not been determined. As of September 2015, program officials expect full deployment to be reached by October 2016.

As of October 2015, DEAMS program officials reported that the program did not meet all of its nine key performance targets. Specifically, DEAMS did not meet five performance targets: Balance with Treasury, Accurate Balance of Available Funds, Timely Reporting, Period-End Processing, and Net-Ready. For example, the two operational assessments that were conducted from 2012 to 2014 identified significant weaknesses in three measures of effectiveness and suitability. Further, the Initial Operational Test and Evaluation report identified system performance issues within the DEAMS program, which included change management issues, transaction backlogs, and ineffective reporting tools.

Subsequently, the Air Force Operational Test and Evaluation Center provided 29 recommendations for the Air Force to implement to support the successful fielding of DEAMS Increment 1, 17 of which were documented as being completed, while corrective action for the remaining 12 are still underway. However, according to program documentation, DEAMS must demonstrate measureable improvement by the full deployment decision date of February 2016, in order to avert future schedule delays in its fielding deployment.

Selected Programs Use of IT Acquisition Best Practices Were Mostly Applied for Requirements Management and Fully Applied for Risk Management

According to the Software Engineering Institute’s Capability Maturity Model Integration for Acquisition, an appropriate requirements management involves establishing an agreed-upon set of requirements, ensuring traceability between requirements and work products, and managing any changes to the requirements in collaboration with stakeholders. Likewise, an effective risk management process identifies potential problems before they occur, so that risk-handling activities may be planned and invoked, as needed, across the life of the project in order to mitigate the potential for adverse impacts. Table 2 provides key practices used to comprehensively manage requirements and risk.

All three selected programs implemented IT acquisition best practices for risk management, but requirements management best practices were not consistently implemented by the programs. Table 3 provides a summary of the extent to which requirements and risk management best practices were implemented by each program.

The Army Program Implemented Risk Management Practices, but Did Not Apply Several Requirements Management Practices

The Army implemented all risk management best practices for the TMC program. For example, the risk management plan, dated May 2014, identified risk sources to include, among other things, unclear system requirements, immature technology, and an unstable organizational environment. In addition, program officials analyzed, categorized, and controlled risks using a probability and impact model that considered the risks (from very low to very high) and the potential consequences. The program also used a risk radar tool to track and monitor risks. These are reviewed weekly during staff meetings and updated monthly. Further, TMC’s risk management plan indicated that contingency plans are invoked whenever adjustments to cost, schedule, or performance are required. In taking these and other actions, the TMC program had established and utilized the key risk management practices. Doing so should better position the program to mitigate adverse impacts from potential problems before they occur.

The Army had implemented three requirements management best practices for the program, but did not fully implement two: the practice of managing requirements changes and ensuring that work products are in alignment with requirements. For example, one key practice that the program implemented included the maintenance of the bidirectional traceability tool among requirement. Specifically, program officials utilized a traceability tool used to generate a requirements matrix to track all of its program elements to the requirements. Regarding managing requirements changes, while program officials tracked requirements changes in a database, requirements changes were not always available at the stakeholder level to evaluate the impact and determine the status of requirements changes for all elements of the program. Even though TMC was in the production phase, it relied solely on the functions within its requirements database rather than a requirements management document. This left the program without a formal mechanism to track and ensure project plans, activities, and work products are consistent with defined requirements. Without such a document, the program does not have a formal mechanism to track and ensure that project plans, activities, and work products are consistent with defined requirements.

The Navy Program Implemented Risk Management Practices, but Did Not Apply All Requirements Management Practices

The Navy implemented all risk management best practices for the CAC2S program. For example, the risk management plan assessed risks in terms of their probability and consequence of occurrence. The program also identified and documented risks and had the supporting documentation that included risk and issue register logs, detailed reports, the integrated master schedules, and risk assessments. In addition, the risk management plan established the strategy that included the processes to guide risk mitigation efforts at the lowest, appropriate level. The program’s risk registers, which included risk mitigation steps, were provided by program officials to demonstrate that risk mitigation plans had been implemented for each risk. In taking these and other actions, the program had established and utilized effective risk management practices.

The Navy implemented three requirements management best practices for the program but did not implement two. For example, one key practice implemented was managing requirements changes. Specifically, the program office demonstrated the ability to effectively manage changes to requirements as they evolved during the project. Program officials did this by documenting the alignment of requirements to its respective requirements changes, maintaining a history of requirements changes with rationale explaining the change request within its configuration documentation, and publishing requirements data using a database tool. However, the program did not fully implement the practice of maintaining traceability among requirements and work products to ensure that work products were in alignment.

Further, with regard to the program’s bidirectional traceability tool, according to the July 2014 traceability tool, 25 specifications and 13 capabilities did not map to its respective work products. According to program officials, as of November 2015, 11 of the 25 specifications had been mapped, but due to an oversight, mapping of these specifications were not associated, while the remaining 14 specifications did not map to its respective capability production documentation to demonstrate completeness. According to program officials, 4 capabilities had been recently mapped but the remaining 9 capabilities listed in the capability production documentation did not map the traceability to the respective requirements. According to program officials, the mapping discrepancy of 7 capabilities was attributed to unfunded, obsolete, and programmatic requirements. Furthermore, after notifying the program of the gaps we identified, officials stated that they would take action to ensure mapping of 14 specifications and 2 capabilities to their respective requirements work products would be addressed.

Regarding the alignment of requirements, the program’s requirements management plan had not been updated since May 2009 and software specifications and capabilities were not consistently maintained. According to program officials, the current requirements management plan had been previously reviewed and was determined to be suitable for the purpose of implementing requirements management best practices. According to the Capability Maturity Model® Integration for Acquisition, without a clear linkage between requirements all the lower-level requirements and capabilities, the program may not be effectively managing development efforts in accordance with the most recent requirements. Until work products are updated, the program cannot provide assurance that its requirements are aligned with the most updated work products and is at-risk of potential cost and schedule consequences.

The Air Force Program Implemented Risk Management Practices, but Did Not Apply One Requirements Management Practice

The Air Force implemented all seven risk management best practices for the DEAMS program. To the program’s credit, DEAMS demonstrated great strides in improving its risk management best practices. Since our prior MAIS review, the program has made improvements such as monitoring the status of each risk periodically, and ensuring that risk reports were up to date, which included the status of actions to mitigate risks. Other key practices include defining parameters to analyze and categorize risks, documenting risk, and developing risk mitigation plans in accordance with the risk management strategy, among other areas. In taking these and other actions, the DEAMS program had established and utilized effective risk management practices.

The Air Force had implemented four requirements management best practices, but did not fully implement one requirements management practice, the practice of developing an understanding with providers on the meaning of requirements. For example, the Air Force ensured that project plans and work products were aligned with the most recent requirements. Specifically, the program maintained consistent documentation and oversight of work products, which included an up-to- date requirements management plan, system specifications, and capabilities documentation. However, while the program had established an adjudication process by which requirements were reviewed and approved, and implemented a test methodology to validate requirements prior to production installation, the function to ensure accountability was not working properly.

As such, program officials did not determine whether key requirements were validated during system integration testing prior to deploying software into production, which was released with unresolved issues. Program officials subsequently resolved the issues without any negative impacts. The program office attributed the issue of not fully validating requirements to environmental issues that were considered to be acceptable risks and, subsequently, scheduled production installation in November 2014. Nevertheless, according to the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition, requirements should be analyzed to ensure that established criteria are met so that proper control functions are in place.

All Appropriate MAIS Programs Were Represented on the Federal IT Dashboard, but the Responsible Organization Was Not Provided

Pursuant to its statutory responsibility to analyze, track, and evaluate risks, OMB requires agency CIOs to provide cost, schedule, and risk information for all major IT investments on the IT Dashboard. In addition, the IT Dashboard shows CIO names and photographs who are responsible for investments to increase accountability for IT acquisitions.

As of October 2015, 27 of the 39 DOD MAIS programs were listed on the IT Dashboard. According to DOD officials, and in accordance with OMB policy, 8 MAIS programs that have not been funded in the President’s budget submission are not reported to the Dashboard, as appropriate. In addition, 4 MAIS programs have been designated by DOD as containing national security-sensitive information and were therefore classified and not subject to being reported on the Dashboard. According to DOD CIO and AT&L officials, the 8 unfunded MAIS programs will be reported to the Dashboard after the President’s 2017 budget submission has been finalized.

However, the organization responsible for supervising MAIS acquisition programs—AT&L—is not represented on the Dashboard. Instead, the Dashboard publicly shows DOD’s CIO as the responsible party, pursuant to OMB’s direction, but is not accurate. AT&L has oversight responsibility for the acquisition performance of MAIS programs. In this regard, not only does AT&L supervise department acquisitions and establish its acquisition policies, but as the milestone decision authority for MAIS programs, the Under Secretary or his designee, has overall responsibility for each program. By contrast, the CIO is not involved in managing the performance of the MAIS programs but is responsible for submitting the rating to the Dashboard.

Officials from DOD’s Office of the CIO and OMB’s Office of E- Government and Information Technology told us that they were aware of this inconsistency on the Dashboard but did not think it was a significant issue. Further, the DOD officials stated that since the CIO is involved in the rating process the representation of their office on the Dashboard is sufficient. Nonetheless, since only the DOD CIO is represented on the Dashboard, the public and other users may be unaware that AT&L has overall oversight for the acquisition performance of MAIS programs, minimizing the intended accountability the Dashboard is to provide.

Conclusions

Since MAIS programs account for billions of dollars of DOD’s IT budget, it is important that the required critical change reports are timely so Congress has the necessary information to make budgetary and oversight decisions. While the reports contained the required elements, many were not submitted in a timely manner, potentially hampering Congress’ ability to make informed decisions.

Further, all three selected programs implemented IT acquisition best practices for risk management and implemented most practices for requirements management. While this is a significant achievement, improvements can be made in managing requirements. Among other things, programs were operating without a current requirements management plan that was considered to be acceptable risks. Managing requirements effectively is especially necessary since MAIS programs are intended to help the department sustain its key operations.

Finally, there is a lack of accountability for AT&L on the IT Dashboard. While OMB intended for accountability by requiring that major investments show agency CIOs as responsible, it did not consider that DOD’s AT&L is the responsible party for oversight of the acquisition performance of MAIS programs. Since the Dashboard does not reflect that AT&L has such responsibility, there is decreased public accountability.

Recommendations for Executive Action

To help improve the management of MAIS programs, we are making six recommendations that:

The Secretary of Defense examine the MAIS critical change reporting process to identify root causes for delays and implement corrective actions for the timely delivery of critical change reports.

The Secretary of Defense develop a mechanism for monitoring whether MAIS programs with late reports are restricted from obligating funds and in turn ensuring compliance with the Antideficiency Act.

The Secretary of the Army direct the TMC program manager to develop a requirements management plan to document and manage its requirements process.

The Secretary of the Navy direct the CAC2S program manager to identify weaknesses in the requirements traceability process and take corrective actions to manage the traceability of requirements to the respective lower-level requirements, and periodically evaluate work products, including the requirements management plan, and update them in accordance with the requirements guidance.

The Secretary of the Air Force direct the DEAMS program manager to address weaknesses in its controls for ensuring that all software requirements are tested and validated before deployment of new software releases.

Director of OMB instruct the Federal CIO to add the Under Secretary of Defense for AT&L as a responsible party to DOD’s MAIS entries on the Federal IT Dashboard website, alongside the CIO, to publicly disclose the responsible party for the acquisition performance management of MAIS programs.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD and OMB. We received written comments from DOD’s Acting Principal Deputy Assistant Secretary of Defense for Acquisition, which are reprinted in appendix III. In its comments, the department concurred with all five recommendations to improve oversight, IT acquisition practices, and tools used to manage MAIS programs.

In e-mail comments, an official from OMB’s audit liaison group stated that OMB’s Office of E-Government and Information Technology does not agree with the recommendation to add AT&L to the IT Dashboard as a responsible party for MAIS programs but would work with DOD to address it. The official did not provide a rationale for this position or explain how OMB would work with DOD.

Nonetheless, we continue to believe there is a lack of transparency and accountability for AT&L on the IT Dashboard. The IT Dashboard publicly shows DOD’s CIO as the responsible party, pursuant to OMB’s direction. However, AT&L has oversight responsibility for the acquisition performance of MAIS programs. In this regard, not only does AT&L supervise department acquisitions and establish its acquisition policies, but as the milestone decision authority for MAIS programs, the Under Secretary or his designee, has overall responsibility for each program. The DOD CIO is not involved in managing the performance of the programs and is only responsible for submitting the rating to the Dashboard. We believe that adding AT&L to the IT Dashboard would increase public accountability and leadership transparency for the acquisition management of MAIS programs.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Secretary of the Air Force, the Secretary of the Army, the Secretary of the Navy, the Office of Management and Budget, and other interested parties. This report also is available at no charge on the GAO website at http://www.gao.gov.

Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or ChaC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The National Defense Authorization Act for Fiscal Year 2012 includes a provision that we select, assess, and report on selected Department of Defense (DOD) major automated information system (MAIS) programs annually through March 2018. In addition, Senate Report 113-176 accompanying S. 2410 includes a provision that we evaluate DOD’s implementation of statutory reporting requirements for MAIS programs experiencing a critical change. Our objectives for this report were to (1) evaluate DOD’s implementation of statutory reporting requirements for MAIS programs experiencing a critical change; (2) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates, and met performance targets; (3) assess the extent to which selected MAIS programs have used key IT acquisition best practices, including risk management; and (4) determine the extent to which MAIS programs are accurately represented on the Federal IT Dashboard.

To evaluate DOD’s implementation of statutory reporting requirements for MAIS programs experiencing a critical change—a schedule delay of 1 year or more, a full life-cycle cost increase of 25 percent or more over the original estimate, or a change that will undermine the system’s ability to perform as intended—we collected and analyzed information about critical changes from December 2008 to June 2014 and their corresponding reports. We assessed whether DOD had met reporting requirements by reviewing MAIS critical change reports and supporting documentation to determine if they included required written certifications stating that: the automated information system or IT investment to be acquired is essential to the national security or to the efficient management of the DOD; there is no alternative to the system or IT investment which will provide equal or greater capability at less cost; the new estimates of the costs, schedule, and performance parameters have been determined, with the concurrence of the Director of Cost Assessment and Program Evaluation, to be reasonable; and the management structure for the program is adequate to manage and control program costs.

We did not look at the quality of the assessments and estimate done. We reviewed the reports to determine if the particular elements were included. We then summarized the number of elements addressed by program.

In order to determine if the critical change reports met or exceeded the 60-day requirement, we collected all MAIS reports and compared the dates within them. We subtracted the date in which the reports were delivered to Congress via letter from the date in which the program manager provided the quarterly report to the senior DOD official responsible for the program in order to determine the number of days that had lapsed. Additionally, we rearranged the programs by number of days (from greatest to least) it took to deliver the critical change reports. We noted how many programs took more than 100 days, and 200 days to deliver the reports. We also interviewed DOD officials in order to ask their opinion on the length of time it takes to deliver a critical change report to Congress.

We interviewed DOD officials responsible for the quality of the data and assessed their procedures for maintaining its accuracy and completeness. In addition, we examined the data for outliers or others extraordinary items. Based on these procedures, we have concluded that these data are sufficiently reliable for our purposes.

Additionally, in order to determine whether DOD is tracking the obligation of funds during a program’s critical change, we interviewed DOD officials in order to determine what processes and tools they had in place to ensure that funds were not being obligated.

To address the second and third objectives, we used DOD’s official list of 39 MAIS programs, as of February 25, 2015, to establish the basis for selecting the MAIS programs that were used to assess objectives two and three. We used the criteria below to select three MAIS programs.

Any programs that were used in the prior two reviews should be excluded.

Any programs that are fully deployed or cancelled should be eliminated from consideration.

The program should not be new to the MAIS list; otherwise, there may not be sufficient acquisition activity and documentation to evaluate.

The program must have a baseline in order to have a reference point for evaluating cost and schedule performance characteristics.

The program cannot be a National Security Agency program.

We included one program from each military department—Army, Air Force, and Navy in order to diversify the portfolio. Thus, we excluded any DOD-wide programs.

We selected programs that had the lowest relative ratings on the Federal IT Dashboard as of April 2015.

We preferred that the program be complex (e.g., integration across domains, global, critical to battle operations, etc.), rather than an upgrade.

We preferred to select programs with funding profiles that are significant when compared to the rest of the portfolio.

We considered issues identified from credible sources of information, such as Defense Acquisition Management Information Retrieval online resources, IT Dashboard ratings, etc.

We filtered the original list of MAIS programs using the criteria above. Based on this filtering, we chose the following systems: the Air Force’s Defense Enterprise Accounting and Management System-Increment 1 (DEAMS Increment 1), the Army’s Tactical Mission Command (TMC), and the Navy’s Common Aviation Command and Control System Increment 1 (CAC2S Increment 1).

To address the second objective, we analyzed and compared each selected program’s first acquisition program baseline cost estimate to the latest life-cycle estimate to determine the extent to which planned program costs had changed. Similarly, to determine the extent to which these programs changed their planned schedule estimates, we compared each program’s first acquisition program baseline schedule to the latest schedule. We relied on the thresholds established by statute to describe the amount of any deviation (i.e., significant or critical) that each program’s latest life-cycle cost and schedule estimates experienced from the first acquisition program baseline.

To determine whether the selected programs met their performance targets, we compared program and system performance targets against actual performance data in test reports and program management briefings. We reviewed the results of operational assessments and program evaluations conducted on the systems. We also reviewed additional information on each program’s cost, schedule, and performance, including program documentation, such as DOD’s MAIS annual and quarterly reports; information from the Office of Management and Budget’s (OMB) IT Dashboard; acquisition program baselines; monthly status briefings; system test reports; and our prior reports. We also interviewed program officials from each of the selected MAIS programs to obtain additional information on cost, schedule, and performance. We provided our assessments to the program management offices of each selected program for comment. We aggregated and summarized the results of these analyses across the programs, as well as developed individual profiles for each program (see appendix II).

To address the third objective, we analyzed each selected program’s IT acquisition documentation and compared it to key requirements management and risk management best practices—including Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (CMMI- ACQ) practices—to determine the extent to which the programs were implementing these practices. In particular, the key requirements management best practices we reviewed were: develop an understanding with the requirements providers on the meaning of the requirements, obtain commitment to requirements from project participants, manage changes to requirements as they evolve during the project, maintain bidirectional traceability among requirements and work, and ensure that project plans and work products remain aligned with requirements.

Specifically, we analyzed program requirements documentation, including requirements management plans, requirements traceability matrices, requirements change forms, technical performance assessments, and requirements board meeting minutes. Additionally, we interviewed program officials to obtain additional information about their requirements management practices.

Additionally, we reviewed the following key risk management best practices: determine risk sources and categories; define parameters used to analyze and categorize risks and to control the risk management effort; establish and maintain the strategy to be used for risk management; identify and document risks; evaluate and categorize each identified risk using defined risk categories and parameters, and determine its relative priority; develop a risk mitigation plan in accordance with the risk management monitor the status of each risk periodically and implement the risk mitigation plan as appropriate.

Specifically, we analyzed program risk documentation, including monthly risk logs and reports, risk-level assignments, risk management plans, risk mitigation plans, and risk board meeting minutes. Additionally, we interviewed program officials to obtain additional information about their risks and risk management practices.

To address the fourth objective, we used DOD’s official list of MAIS programs, as of February 25, 2015. These programs were used as basis to determine whether programs were reported to the Federal IT Dashboard. To do so, we:

Exported the IT portfolio program data reported to the Federal IT Dashboard by DOD in fiscal year 2015, and compared it to the 39 programs on DOD’s official list of MAIS programs.

For those programs that were found to not be reported on the Dashboard, we met with agency officials from the Office of the Chief Information Officer (OCIO) and the Under Secretary of Defense (Acquisition, Technology, and Logistics) (AT&L) to determine the reasons for not reporting.

We also interviewed OCIO officials to obtain information on the processes used by the OCIO when reviewing programs for IT Dashboard updates. Further, we interviewed officials at OMB to obtain their views on representing the CIO as the sole party responsible for programs reported to the IT Dashboard was accurate. Specifically, we discussed DOD’s unique structure that AT&L is responsible for the acquisition performance of MAIS programs but is a separate organization from the OCIO.

We conducted this performance audit from April 2015 to March 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Profiles of Selected MAIS Programs

This section contains profiles of the three selected major automated information system (MAIS) programs for which we determined whether they had changed their planned cost and schedule estimates and met performance measures. Each profile presents data on the program’s purpose and status, its latest cost and schedule estimates compared to the first acquisition program baseline (where established), as well as system performance data.

The first page of each two-page profile contains a description of the program’s purpose and a figure that provides a comparison of the program’s first acquisition program baseline to the program’s latest schedule. The years depicted on the figure represent calendar years, and the milestones represent the program’s best estimates of dates for those milestones. The program’s start represents the date that program officials reported that they first started work on the program.

The first page also provides (1) essential program details, such as the name of the prime contractor, the total number of active contractors— which includes the prime contractor—and any other contractors (and in some cases subcontractors) supporting the program; (2) program costs (in then-year dollars), comparing the program’s latest life-cycle cost estimate (separated into acquisition and operations and maintenance costs) to its first acquisition program baseline (subsequent acquisition program baselines that may have been established are not identified), (3) locations to which the system will be deployed; and (4) a summary of the cost, schedule, and performance of each program, which is further discussed on the second page of the profile. The symbols, denoted by arrows or filled circles, included in the summary box on the first page of each profile and in the headings on the second page represent whether a program’s cost estimate had increased (^), decreased (V), or stayed within (i.e., not to exceed threshold) planned cost estimate (i.e., () and whether the program’s schedule estimate had slipped (>),been accelerated to meet milestones earlier than planned (<), or stayed within (i.e., not to exceed threshold) planned schedule estimate () of meeting milestones.

The second page of each profile provides detailed information on each program’s status, costs, schedule, and performance.

In the status section, we discuss recent and upcoming milestones and events for each program. In the cost section, we identify the extent to which the program’s life- cycle cost estimate has changed from its first acquisition program baseline, as well as the causes for any changes identified. In the schedule section, we discuss the extent to which the program’s schedule has changed from its first acquisition program baseline, and the causes for any schedule changes identified. In the performance section, we identify the extent to which each program has met its established measures, as well as discuss the results of system performance tests. These performance ratings represent a point-in-time assessment as reported by the program. System performance targets were rated as “met” when (1) system tests were passed with no deficiencies or limitations, (2) the program fully met all of its key performance parameters, or (3) a program had addressed all deficiencies or limitations that were identified during system tests. System performance was rated as “not fully met” when a program either (1) did not fully pass system testing and was still in the process of addressing the deficiencies or limitations identified during system testing or (2) did not pass system testing and subsequently removed the problematic functionality from the system in order to pass subsequent system tests, instead of fixing the problematic functionality and keeping it in the planned release of the system.

Army Tactical Mission Command (TMC)

TMC is a suite of products—comprised of hardware and software equipment and elements—that are intended to provide the Army commanders and their staff with mission command capabilities, such as real-time situational awareness and a user-defined common operational picture. TMC products are fielded worldwide and are intended to support decision-making, planning, rehearsal, and execution management. TMC is now engaged in transitioning to the web-based Command Post Computing Environment. One key element of this new environment — known as Tactical Applications—is aimed to minimize administrative burdens on the user, and simplify the overall Mission Command collaborative experience.

All of the products included in TMC are post-development and in production. The program continues to field equipment and perform a technical refresh of the hardware and software in the field. The program is also working to resolve sustainment metric issues and updating their life- cycle sustainment plan.

Exceeded Planned Cost Estimate (^)

TMC’s planned total life-cycle cost estimate has increased by 19 percent from the program’s first acquisition program baseline estimate of approximately $1.97 billion. Specifically, as of January 2016, the life-cycle cost estimate was approximately $2.34 billion. Program officials reported that the increased costs were attributed to research and development testing and evaluation cost estimate breach that was reported to Congress. The Army’s TMC program estimated program development cost increased by 45 percent over the original estimate due to program scope changes derived from the realignment of Command Post of the Future as a foundation for Mission Command Collapse, the integration of Personalized Assistant that Learns, and the incorporation of future force requirements.

Stayed within Planned Schedule Estimate ()

As of January 2016, the program had experienced a 3 month slippage in its full deployment date compared to its first acquisition program baseline of September 2018. The slippage was within the pre-established threshold allowance to account for minor shifts in program schedule. Program officials stated that, although the Command Post of the Future product is 95 percent fielded and is on schedule to reach full deployment by December 2018, continued support of the Command Post Computing Environment is needed beyond fiscal year 2019. Program officials considered the slippage to be of low risk and will continue to operate within the planned schedule estimate.

As of January 2016, the TMC program met all three of its key performance parameters, including net-centric military operations, disseminate orders with future Army and Joint C2 systems, and displaying unified information on subject matters, such as friendly and enemy forces.

Navy Common Aviation Command and Control System (CAC2S) Increment 1 CAC2S is an integrated and coordinated modernization effort for the equipment of the Marine Air Command and Control System and is intended to provide enhanced capability for three defense centers to support aviation employment in joint, combined, and coalition operations. CAC2S provides the tactical situational display, information management, sensor and data link interface, and operational facilities for planning and execution of Marine Aviation missions within the Marine Air Ground Task Force. It is intended to replace existing aviation command and control equipment from 12 legacy systems. CAC2S Increment 1 will eliminate the Air Command and Control systems and will capability for aviation combat direction and air defense functions by providing a single networked system.

CAC2S Increment 1, which comprises of two phases, is currently in post- milestone C where the phase 2 system is now in development. CAC2S’s current work consists of developmental testing and the production of the limited deployment units in preparation for the March 2016 Initial Operational Test and Evaluation. To help achieve its goals of a successful Initial Operation Test and Evaluation, a phase 2 milestone C decision was authorized in February 2015 for CAC2S to procure four limited deployment units. As of October 2015, CAC2S had delivered all four limited units to support current developmental testing. Production of the limited deployments units are on schedule and planning activities are underway. Regarding the developmental testing, program officials anticipate favorable test results after subsequent software enhancements had been made to address software concerns identified in prior developmental testing.

Exceeded Planned Cost Estimate (^)

As of October 2015, CAC2S’s life cycle cost estimate was $2 billion, which was about a 477 percent increase from its first acquisition program baseline estimate of $347 million established in August 2000. As previously reported, factors that attributed to the cost increase were early challenges in estimating costs due to program scope growth and restructuring. According to program documentation, operations and support expenditures of approximately $1.6 billion for the production of milestone C had been carried over into the program’s total life-cycle cost estimate. However, since our previous report, program documentation indicated that improvements to the cost position are being made. Specifically, the milestone C service cost position, dated February 2015, produced a cost avoidance of $54.4 million compared to its 2010 cost assessment. As of October 2015, the program’s latest life-cycle estimate relative to its November 2010 production acquisition program baseline cost estimate had decreased about 19 percent. Program officials attributed this decrease due to the program embracing the DOD Better Buying Power initiatives that benefitted from competitive market forces that drove down cost.

Exceeded Planned Schedule Estimate (>)

As of October 2015, CAC2S Increment 1’s estimated full deployment date was March 2022, which represented a 13 year and 9 month schedule slip from the program’s first acquisition program baseline schedule estimate. As previously reported, factors that attributed to the schedule delay included the addition of new requirements and program restructure. Program officials stated that subsequent to our prior report, the program has been executing in accordance within its approved schedule. As of October 2015, the estimated milestone C phase 2 was delayed by 6 months from the program’s production acquisition program baseline schedule but, as stated above, achieved its milestone. Program officials attributed this delay, in part, to administrative factors, which included the review and approval process of getting signature approval. CAC2S successfully achieved milestone C phase 2 approval in February 2015 but the acquisition decision memorandum had not been signed until March 2015.

As of October 2015, CAC2S program documentation reported that it was meeting both of its key performance parameters related to net-ready and data fusion. Program officials stated that, during testing of the program’s key performance parameters, its net-ready and data fusion performance targets were both met, while many attributes for the data fusion key performance parameter were consistently above the threshold for being met.

Air Force Defense Enterprise Accounting and Management System (DEAMS) Increment 1 The DEAMS Increment 1 program is intended to provide the Air Force with the entire spectrum of financial management capabilities, including collections; commitments and obligations; cost accounting; general ledger; funds control; receipts and acceptance; accounts payable and disbursement; billing; and financial reporting DEAMS is also intended to be a key component of DOD’s solution for achieving fully-auditable financial statements by September 30, 2017, as required by the National Defense Authorization Act for Fiscal Year 2010.

As of November 2015, the DEAMS program is working to achieve full deployment decision by February 2016. In August 2015, the initial operational test and evaluation report, conducted by the Air Force Operational Test and Evaluation Center, indicated a number of findings requiring remediation prior to the February 2016 full deployment decision. Nevertheless, DEAMS was granted a limited deployment decision, but the program experienced a significant change as a result of breaching the full deployment decision threshold for a timing issue only. DEAMS current work efforts consist of deployment to new users and remaining 35 sites, capability development for deployment, training new users, and resolving initial operational test and evaluation findings.

Exceeded Planned Cost Estimate (^)

As of October 2015, DEAMS’s latest life-cycle cost estimate was about $1.56 billion, which was about a 9 percent increase from its first acquisition program baseline estimate of approximately $1.43 billion— established in February 2012. Program officials attributed this increase, in part, to program scope growth due to addition of requirements from increment 2 and the addition of a second Oracle software upgrade projected for 2021.

Exceeded Planned Schedule Estimate (>)

DEAMS experienced a 6 month slippage in its milestone C but successfully attained milestone C approval within the established threshold. Program officials did not provide a rationale for factors that attributed to this delay but maintained that the program operated within the threshold requirements. DEAMS also experienced a 1 year slippage in its full deployment decision date—currently scheduled for February 2016. Program officials attributed this slippage due to findings identified DEAMS’s initial operational test and evaluation report.

Did Not Fully Meet System Performance Targets As of October 2015, DEAMS program officials reported that it did not meet all of its nine key performance parameters. Specifically, DEAMS did not meet five key performance parameters: Balance with Treasury, Accurate Balance of Available Funds, Timely Report, Period-End Processing, and Net-Ready. For example, an initial operational test and evaluation report, listed above, identified system performance issues, which included unstable change management issues, transaction backlogs, and ineffective reporting tools. Subsequently, the Air Force Operational Test and Evaluation Center provided 29 recommendations for the Air Force to implement to support the successful fielding of DEAMS Increment 1, 17 of which were documented as being completed, while corrective action for the remaining 12 are still underway. The program is expected to demonstrate improvement before it will be authorized to be deployed to all users.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, the following staff also made key contributions to this report: Eric Winter, Assistant Director; Ronalynn (Lynn) Espedido; Corey Evans; Rebecca Eyler; Franklin Jackson; Kate Nielsen; John Ortiz; Kathleen Sharkey; and Jeanne Sung.