Army Networks: Opportunities Exist to Better Utilize Results from Network Integration Evaluations
Released: 2013-08-22T00:00:00

Letter

Background

As the Army transitions away from major wartime operations, it faces fiscal constraints and a complex and growing array of security challenges. The Army will be smaller and senior leaders recognize that the core of a smaller yet still highly capable force is having a capable tactical information network.

Over the last decade, the Army focused most of its decisions to field network improvements on supporting operations in Iraq and Afghanistan, an effort that was both expensive and time consuming. The Army did not synchronize the development and fielding efforts for network technologies. Funding and time lines for network-related programs were rarely, if ever, aligned. The Army fielded capabilities in a piecemeal fashion and the user in the field was largely responsible for integrating them with existing technology.

In December 2011, Army leaders finalized the Network-enabled Mission Command Initial Capabilities Document, a central document that describes the essential network capabilities required by the Army as well as scores of capability gaps. These capabilities support an Army mission command capability defined by a network of command posts, aerial and ground platforms, manned and unmanned sensors, and dismounted soldiers linked by an integrated suite of mission command systems. A robust transport layer capable of delivering voice, data, imagery, and video to the tactical edge (i.e., the forward battle lines) connects these systems.

To achieve the objectives of its network modernization strategy, the Army is changing the way it develops, evaluates, tests, and delivers networked capability to its operating forces, using an approach called capability set management. A capability set is a suite of network components, associated equipment, and software that provides an integrated network capability. A requirement is an established need justifying the allocation of resources to achieve a capability to accomplish military objectives. Instead of developing an ultimate capability and buying enough to cover the entire force, the Army plans to buy only what is currently available, feasible, and needed for units preparing to deploy. Every year, the Army will integrate another capability set that reflects changes or advances in technology since the previous set. To support this approach, the Army is implementing a new agile process that identifies capability gaps and solicits solutions from industry and government to evaluate during the NIEs.

NIEs are a significant investment. Since 2011, the Army has conducted five of them, and has projected the cumulative cost of the events at $791 million. The Army conducts NIEs twice a year. Each NIE typically involves around 3,800 soldiers and 1,000 vehicles, and up to 12,000 square kilometers of territory, and approximately 6 weeks in duration. The two categories of the key participating systems during the NIEs are Systems under Test (SUT) and Systems under Evaluation (SUE), and each is subject to differing levels of scrutiny.

SUTs are from an ongoing acquisition program (sometimes referred to as a program of record) that are formally determined to be ready for operational testing in order to inform an acquisition decision. This operational testing is subject to review and is conducted with the production or production-like system in realistic operational environments, with users that are representative of those expected to operate, maintain, and support the system when fielded or deployed.

SUEs are provided by either industry or the government. They are either (1) developing capabilities with sufficient technology, integration, and manufacturing maturity levels to warrant NIE participation; or (2) emerging capabilities that are seen as next generation war-fighting technologies that have the potential to fill a known gap or improve current capabilities. SUEs are not subject to formal test readiness reviews, nor the same level of testing as the SUTs. SUEs are operationally demonstrated and receive a qualitative user evaluation, but are not operationally tested and are not the subject of a formal test report (as SUTs are).

Aside from their role in the agile process, NIEs also provide the Army with opportunities for integration, training, and evaluation that leads to doctrine, organization, training, materiel, leadership and education, personnel, and facilities recommendations; and the refinement of tactics, techniques, and procedures related to the systems tested.

The Army believes that traditional test and evaluation processes frequently result in fielding outdated technologies and expects to improve on those processes through the NIEs. The Army’s test community members, including the Brigade Modernization Command (BMC) and the Army Test and Evaluation Command (ATEC), conduct the testing during the NIEs. The BMC is a headquarters organization within the Training and Doctrine Command. It has an attached operational 3,800-soldier brigade combat team dedicated to testing during the NIEs. BMC soldiers use systems during the NIE in simulated combat scenarios for testing and evaluation purposes, resulting in qualitative evaluations based on their observations. The BMC also recommends whether to field, continue developing, or stop developing each solution and to improve the integration of capabilities into deploying brigades. ATEC has overall responsibility for the planning, conduct, and evaluation of all Army developmental and operational testing. ATEC also produces a qualitative assessment of the overall performance of the current capability set of network equipment.

Two test offices within the Office of the Secretary of Defense that help inform Defense Acquisition Executive decisions also provide oversight on testing related to major defense acquisition programs. The Director, Operational Test and Evaluation (DOT&E) provides oversight of operational testing and evaluation for SUTs. The Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DT&E) provides oversight of developmental testing that precedes operational testing of SUTs. DOT&E and DT&E roles are limited to the SUTs selected for operational testing during the NIEs.

Test and evaluation is a fundamental aspect of defense acquisition. DOD, under its Defense Acquisition System, requires the integration of test and evaluation throughout the defense acquisition process to provide essential information to decision makers; assess attainment of technical performance parameters; and determine whether systems are operationally effective, suitable, survivable, and safe for intended use. Testers generally characterize test and evaluation activities as either developmental or operational. Developmental testing is a generic term encompassing modeling and simulation and engineering type tests that are used to verify that design risks are minimized, that safety of the system is certified, that achievement of system technical performance is substantiated, and that readiness for operational test and evaluation is certified. The intent of developmental testing is to demonstrate the maturity of a design and to discover and fix design and performance problems before a system enters production.

Operational testing is a field test of a system or item under realistic operational conditions with users who represent those expected to operate and maintain the system when it is fielded or deployed. Specific operational tests include limited user tests, initial operational tests, and customer tests. Before operational tests occur for major acquisition programs, DT&E completes an independent Assessment of Operational Test Readiness. Each Assessment of Operational Test Readiness considers the risks associated with the system’s ability to meet operational suitability and effectiveness goals. This assessment is based on capabilities demonstrated in developmental testing. The Defense or Component Acquisition Executive considers the results of the Assessment of Operational Test Readiness, among other inputs, in making decisions on a major acquisition program proceeding to operational testing.

Army Not Taking Full Advantage of Potential Knowledge from NIEs

The Army has made steady improvements in the NIE process since its inception and the evaluations continue to give the Army useful information and helpful insights into current and emerging networking capabilities. However, some resulting Army decisions are at odds with knowledge produced during the NIEs. Most importantly, despite poor operational test results for a number of SUTs during the NIEs, the Army has sought approval to buy additional quantities and field several major networking systems. While many of the SUEs received favorable reviews, the Army lacked a strategy that addresses a number of procurement barriers—such as funding availability and requirements—when it began the NIE process, which precluded rapid procurement of successful SUEs. Additionally, as we reported previously, the Army has not yet tapped into the potential to use the NIE to gain insight into the effectiveness and performance of the overall tactical network.

NIEs Provide Several Advantages and Produce Volumes of Potentially Useful Information

To date, the Army has conducted five NIEs, costing an average of $158 million to plan and execute. Through those five NIEs, the Army has operationally tested 19 SUTs and evaluated over 120 SUEs.

NIEs have helped the Army in a number of ways. The NIEs allowed the Army to formulate a network architecture baseline that will serve as the foundation upon which the Army plans to add networking capabilities in the future; evaluate industry-developed systems that may help address Army- identified capability gaps in the future; integrate the new capability sets into operational units and to create new tactics, techniques, and procedures for using the new systems in operations; and provide soldiers with an opportunity to both provide input into the designs of networking systems and to integrate the systems before the Army fields them to operational brigades.

According to Army officials, testing during each NIE generates a large volume of potentially useful information. There are detailed operational test and evaluation reports for each of the SUTs, user evaluations for each of the SUEs, an integrated network assessment of the current capability set, and general observations on the NIE event itself.

The DOT&E has reported observations of the NIEs in its fiscal years 2011 and 2012 annual reports, including an overall assessment, operational scenarios and test design, threat information operations, and logistics. According to DOT&E, the intended NIE objective to test and evaluate network components together in a combined event is sound, as is the opportunity to reduce overall test and evaluation costs by combining test events. NIEs also offer the opportunity for a more comprehensive evaluation of a mission command network instead of piecemeal evaluation of individual network components. In addition, the DOT&E generally reported overall improvements in the execution of the NIEs, realistic and well-designed operational scenarios, and improvements in threat information operations.

ATEC, in addition to preparing operational test reports for specific systems, also prepares an integrated network assessment after each NIE. The reports attempt to characterize how well the current capability set performed with respect to several essential capabilities the Army needs for improved mission command. Based on the performance characterizations presented in the available reports for all NIEs, it appears the Army is making progress in improving its networking capabilities. For instance, the integrated network assessments for NIEs 12.2 and 13.1 cited improvements in an essential capability called network operations. These reports also showed improvements in the common operating picture, which is a capability that enables the receipt and dissemination of essential information to higher echelon command posts. As the Army has modified the reports to improve how they present both capability set performance and essential capabilities, the reports have become tools that are more useful for decision makers.

Army Accepting Risk by Buying and Fielding Systems with Poor Operational Test Results During the NIEs

Four SUTs that the Army plans to buy and field as part of capability set 13—Warfighter Information Network-Tactical (WIN-T) Increment 2, Joint Tactical Radio System Manpack Radio, Joint Tactical Radio System Rifleman Radio, and Nett Warrior—have demonstrated continued poor performance and/or reliability in both developmental tests before NIEs and operational tests during the NIEs. According to the DOT&E, system development best practices dictate that a system should not proceed to operational testing until it has completed developmental testing and corrected any identified problems. To address these problems, the Army has taken steps to implement design changes and schedule additional testing to verify performance after it has implemented those changes. However, in doing so, the Army faces the risk of making system design changes during the production phase or fielding systems with less than required performance or reliability.

Two of these SUTs performed poorly during developmental testing. Developmental testers, through their Assessment of Operational Test Readiness reports, recommended that the Manpack Radio and the Rifleman Radio not proceed into operational testing. Despite these recommendations, the Army proceeded with initial operational testing for these systems during NIEs while reclassifying the participation of other systems as either limited user tests or customer tests. The outcomes were predictably poor, according to DOT&E. See table 1 for operational test results from ATEC and DOT&E reports.

In its 2012 annual report, DOT&E pointed out that proceeding to operational testing only confirmed the deficiencies identified in developmental testing. For example, the WIN-T Increment 2 system’s reliability was troublesome enough in a limited user test to warrant a reduction in the reliability requirement. However, WIN-T Increment 2 was unable to meet the reduced requirement. The Rifleman Radio also demonstrated poor reliability during developmental testing in 2011 and even worse reliability in operational testing due to the enhanced stress of an operational environment. The DOT&E stated in its 2012 annual report that, according to system development best practices, the Army should not proceed to an Initial Operational Test and Evaluation with a system until it has completed developmental testing and the program has corrected any identified problems. Otherwise, the Army may conduct costly operational tests that simply confirm developmental testing conclusions about poor system performance and reliability rather than taking action to fix system shortfalls. Further, DOT&E’s 2012 annual report was critical of the Army’s NIE schedule-driven approach, which elevates meeting a schedule above adequately preparing a system to achieve success in operational testing. An event-driven approach, conversely, would allow systems to participate in a test event after the systems have satisfied certain criteria. Under the Army’s schedule-driven approach, the NIEs are held twice a year and SUTs must align their operational testing to coincide with the next available NIE. An event driven-approach—versus a schedule-driven approach—is the preferred method of test scheduling. Using a schedule-driven approach can result in fielding systems that do not provide adequate utility for soldiers and require costly and time-consuming modification in theater.

Army Working to Address Performance and Reliability Issues Identified During NIE

In light of poor operational test results during previous NIEs, the Army now must pay for and conduct additional, unanticipated, tests to improve system performance and reliability. The extent to which the additional tests corrected all of the identified problems is unknown at this time as the Army awaits the results of the operational testing conducted at the most recent NIE. Ideally, the Army would demonstrate greater levels of operational effectiveness and suitability prior to making production and fielding decisions. Both GAO and DOT&E have acknowledged the risks of proceeding through testing, and to procurement, with systems that perform poorly. Such systems often require design changes that frequently happen when systems are already in production, which can be more costly and technically challenging. Table 2 summarizes the additional activities required of selected systems.

In addition to the unplanned testing summarized in Table 2 above, several systems have operational test and evaluation events scheduled. See table 3.

Despite the poor test results and unplanned activities intended to improve SUT performance, the Army has begun fielding SUTs for capability set 13, including WIN-T Increment 2, Joint Tactical Radio System (JTRS) Manpack radio, Rifleman Radio, and Nett Warrior. Without disputing the test findings and their implications, Army leadership indicates that this equipment addresses critical capability shortfalls and operational needs by providing some level of capability that is otherwise unavailable. For example, most deployed units previously had no or very limited capabilities other than voice communications. Consequently, the Army believes it is urgent to modernize deploying units as quickly as possible, with the equipment in capability set 13.

The Army’s approach carries risk. DOT&E has indicated that the principal way of operating a less reliable system is to invest more in recurring maintenance, which will enable the system to function, but will add to the program’s life-cycle costs and increase its logistical support needs. As a result, the Army will likely have to work with a system that is less reliable than originally envisioned, and develop a new life-cycle cost estimate that reflects the added costs associated with the increased contractor support to keep this less reliable system operating. In addition, ATEC officials state that the negative impact of an individual system falling short of its reliability target is magnified in the capability set. This approach can result in fielded systems that do not provide adequate utility for soldiers and require costly and time-consuming modification in theater as well as additional testing. Our past work as well as reports from DOT&E and DT&E have all found benefits from adequate developmental testing prior to fielding to prove system performance.

Army Buying Few Systems under Evaluation, Despite Positive Soldier Feedback

Since the first NIE in 2011, the Army has evaluated more than 120 SUEs from both industry and government, many of which have received positive reviews and recommendations for fielding from the soldiers. However, the Army has been unable to buy many of these systems because it did not have a strategy in place to rapidly buy promising technologies. Army officials explained that existing DOD acquisition processes would not allow the Army to quickly acquire SUEs that could immediately address networking capability gaps. Even so, Army officials did not develop alternative acquisition approaches before they began the NIE process. It is unclear how long industry will continue to participate in the NIEs if the Army is unable to begin buying systems. As discussed later in this report, the Army has now developed new approaches to address barriers to its ability to quickly buy and field SUEs that have successful demonstrations during the NIEs.

Many SUEs have received positive reviews from soldiers at the NIEs— about five out of every six SUEs were recommended for fielding, field and continue development, or potential for follow-on assessment. Table 4 shows the range of soldiers’ recommendations.

To date, the Army has decided to buy only three SUEs—a company command post, which is a collection of capabilities that enhances a company commanders’ ability to plan, execute, and monitor operations; a touch screen-based mission command planning tool; and an antenna mast. The Army will field only one of these systems in capability set 13— the company command post. While Army officials tell us they would like to buy more systems, a number of factors—such as available funds, deployment schedules, system maturity, and requirements—determine which systems they can buy and when they can buy them. Because it did not have a strategy during the NIEs to address these factors, the Army has been limited in its ability to buy successfully demonstrated SUEs.

The Army expects industry participants to fund fully their own involvement and initial participation in the process and NIEs, which can be a costly endeavor. Army officials have said it can cost up to $250,000 for an interested contractor to provide a whitepaper for consideration. These whitepapers, which interested contractors submit to the Army in response to a sources sought notice, are the industry contractor’s first opportunity to explain both their system and how it addresses a particular capability gap. The Army releases a sources sought notice to industry to solicit candidate commercial solutions for network/non-network capability gaps and the notice informs potential responders of evaluation criteria and subsequent NIE participation criteria. Participation in later phases of the agile process, and ultimately the participation in a NIE can cost the contractor an estimated million dollars, depending on the system the Army is evaluating. Because of the limited number of successfully demonstrated SUEs that the Army has purchased to date, and the cost associated with industry participation, there is concern that industry may lose interest. This could be especially problematic for the Army’s agile process which, according to the Army, is heavily dependent on industry participation for success. Army officials remain confident in the continued support of industry, but the depth and longevity of this support is unclear at this time.

Army Not Using NIE to Evaluate Overall Network Performance and Effectiveness of Investments

While the NIEs are a good source of knowledge for the tactical network as a whole, the Army has not yet tapped into that potential. In January 2013, we reported the Army had not yet set up testing and associated metrics to determine how network performance has improved over time,which limited the evaluation of the cost-effectiveness of its network investments. After completing each NIE, ATEC has provided an integrated network assessment of how well the current capability set enables the execution of the mission command essential capabilities. This qualitative assessment includes only the impact of the current capability set—and not the entire network—on the essential capabilities and does not attempt to evaluate the cost-effectiveness of the current capability set. The Army and DOD consider the fielding of capability set 13 as the initial output from the Army’s network modernization portfolio, but the Army has yet to define fully outcome-based performance measures to evaluate the actual contributions of the capability set. Establishing outcome-based performance measures will allow the Army and DOD to assess progress of network development and fielding and be in a position to determine the cost-effectiveness of their investments in capability set 13. We recommended that, among other things, the Secretary of Defense direct the Secretary of the Army to define an appropriate set of quantifiable outcome-based performance measures to evaluate the actual contributions of capability set 13 and future components under the network portfolio. As discussed later in this report, DOD has started to develop metrics in response to our earlier recommendation.

Army Actions and Additional Opportunities to Enhance the NIE Process

The Army is taking action to correct inefficiencies and other issues based on lessons learned from previous NIEs. The Army is also planning to address potential barriers to rapid procurement of successful SUEs, and DOD has started the process to implement our earlier recommendations on network metrics. Many of the initiatives are in the early stages of implementation so outcomes are not certain. The Army also has an opportunity to work more closely with the test community to further improve NIE execution and results.

Army Is Implementing Corrective Actions to Improve NIE Process and Network Modernization

The Army has identified inefficiencies or less-than-optimal results in its network modernization and the NIE process and has begun implementing corrective actions to mitigate some of them. Table 5 shows some of the major issues identified by the Army and the corrective actions, which are in early stages of implementation.

The Army’s lab-based risk reduction, currently under way, seeks to address concerns over too many immature SUEs sent to past NIEs. Through this initiative, the Army performs technology evaluations, assessments, and integration of vendor systems. Officials test systems individually and as part of an integrated network so that problems can be identified before proceeding to an NIE. In some cases, Army officials identify changes for these systems to increase the likelihood of their success during an NIE, while it drops others when they do not perform well enough in lab testing. Since this effort began, the Army has reduced the number of systems it evaluates during the NIEs, indicating the Army may be making soldiers’ NIE workloads more manageable.

While Army officials acknowledge that lab-based risk reduction does not eliminate all risks, this early evaluation of new systems seems to address some concerns. It may reduce the number of immature systems in the NIE, which could help the Army train soldiers for the new systems. Sending only mature SUEs that have gone through integration testing to NIEs could also help avoid certain test costs.

Additionally, to reduce costs, improve the results of NIEs, and better support rapid fielding of new network capabilities, the test community has reported on several issues requiring corrective action by the Army. Additionally, the testers have also taken actions to help reduce redundancies in test data collection processes, among other things. Implementation of these corrective actions, which testers identified during earlier NIEs, could help prevent negative impacts to NIE testing and modernization. Table 6 describes a number of major issues identified by the test community and corrective actions, which are in early stages of implementation.

Most of the corrective actions to address test community concerns are in early stages of implementation. Below are additional details about the status of a few of the key initiatives.

Army test officials anticipate avoiding $86 million in NIE costs due to implementation of a dozen different efficiency initiatives, including making NIEs more efficient by eliminating duplicative surveys, consolidating data systems, refining SUE test data delivery processes, reducing reliance on contractor data collectors by using military personnel more, and automating data collection. Additionally, BMC officials indicated they intend to incorporate additional testing and reduce the number of soldiers involved in future NIEs to help reduce testing costs. Over time, as the Army conducts NIEs more efficiently, it plans to reduce the number of test personnel, realize commensurate salary savings, and reduce engineering expenses.

Training and guidance for soldiers using new systems during the NIE is another area receiving attention from the test community and the Army. Army test officials reported that there were gaps in soldier training for the SUEs to be evaluated in NIE 13.1. The training issues, in turn, affected the usefulness of the subsequent system evaluations. DT&E officials also expressed concerns about soldier training, and said problems exist in the rehearsal phase of the NIE process. Brigade Combat Team officials said they have also experienced a lack of training resources as they prepare to deploy overseas. According to Army officials, a lack of complete training information, tactics, techniques, and procedures is hampering soldier training on new network systems. That experience was somewhat mitigated, however, by help from soldiers who had used these systems during earlier NIE events. It will be important for the Army to resolve training issues before operational testers qualify systems as fully suitable for combat use following operational testing. Given that operations and support can often comprise about two-thirds of life-cycle costs, a good understanding of these requirements and costs will be necessary for the Army to make well-informed investment decisions for new equipment.

Assessing and using lessons learned from experience can help in planning and implementing future activities. The Army’s efforts to reduce costs and implement corrective actions may take several years; therefore, a continued focus on making NIE processes more efficient and effective, as well as documenting the results of corrective actions would better support the Army’s business case for conducting future NIEs.

Army Efforts to Reduce Barriers to Rapid Procurement of Successful Emerging Technologies

The Army is developing a two-pronged approach to address barriers to its ability to quickly buy and field SUEs that have successful demonstrations during the NIEs. According to Army officials, these barriers included a lack of well-defined requirements for the network system (instead of the more general capability gaps); a lack of funding; and lengthy time frames needed to complete the competitive procurement process. The Army found that the processes for translating capability gaps into requirements, identifying specific funding, and completing a competitive procurement can be very time consuming and challenging.

The Army is now developing a strategy to address these barriers. After the NIE, if the Army decides to buy and field a SUE, the Army plans to align that capability with a suitable existing requirement within an ongoing program of record. The selected program manager would then identify buying options for the capability, including the feasibility of using an existing contract, and would determine whether (1) funding is available, (2) the Army should identify the capability as an unfunded requirement, or (3) the Army needs an above-threshold reprogramming action. The program manager would also determine if the Army can buy and field the capability in the capability set or identify what capability is achievable. Army officials plan to implement this new strategy in the coming months.

In cases where the Army cannot align the successful SUE with an existing program of record, it could develop a new requirement for the system. Army officials have indicated that in a small number of cases, the Army could utilize a directed requirement. The Army generally develops and approves directed requirements to fill urgent needs that the Army believes should be fielded as soon as possible. This allows for essentially bypassing the regular requirements processes, which require additional time to complete.

In addition to this strategy, the Army has developed a new NIE acquisition plan that features an alternative means to buy successful SUEs rapidly.

Under this new plan, the Army is using a combined sources sought notice and a request for proposals approach to better shape requirements and allow for buying SUEs in less time than under normal acquisition processes. With two NIEs per year, the Army will continue to use a sources sought notice to solicit government and industry solutions to broadly defined capability gaps and will assess those solutions during a NIE. Then, the Army will use lessons learned and soldier feedback from the first NIE to validate and refine the requirement and issue a request for proposal for participation in a future NIE. Using a request for proposal differs from using sources sought notices because the request for proposals approach culminates in the award of indefinite-delivery, indefinite-quantity contracts for industry SUEs to participate in a future NIE. Using an indefinite-delivery, indefinite-quantity contract allows the Army to place production orders for industry SUEs following the NIE. The Army released the first request for proposals supporting a NIE on December 20, 2012, to solicit vehicle tactical routers for NIE 14.1. Vehicle tactical routers would allow users and systems located nearby to access networks securely.

For SUEs that already have a defined requirement, the Army plans to issue a request for proposals for participation in one NIE, without using a sources sought notice first. However, Army officials concede that a defined requirement is not usually available prior to the NIE. In those cases, the Army plans to continue issuing sources sought notices for industry proposed solutions that the Army will evaluate during a NIE, as a precursor to issuance of a request for proposals in the future.

The Army expects to comply with current DOD acquisition policy when it decides to buy systems that proceed through the agile process. However, the Army may propose changes to existing policy and processes that inhibit realization of the full benefits of the agile process.

As the Army implements this strategy over the coming months, it will be important to gather information on how well the strategy works and how rapidly the Army can procure and field a SUE after its successful demonstration during an NIE. At the same time, the Army will be in a better position to determine how much of its constrained budget it can devote to the procurement of SUEs. As recommended under internal control standards, it will be important for the Army to establish specific measures and indicators to monitor its performance and validate the propriety and integrity of those performance measures and indicators.This type of information—on how many SUEs the Army can buy and how rapidly—would be helpful for industry as it makes decisions on its future participation in the NIE process.

DOD Has Begun to Define Network Outcome-based Performance Metrics

In our initial report on the Army’s tactical network, we concluded that it will also be important for the Army to assess the cost effectiveness of Moreover, to individual initiatives before and during implementation.facilitate oversight, we concluded that it is important for the Army and DOD to develop metrics to assess the actual contributions of the initial capability set the Army will field in fiscal year 2013 and use the results to inform future investments.

According to a key DOD oversight official reporting on Army networks to the Under Secretary of Defense, Acquisition, Technology, and Logistics, DOD has started work to define quantifiable outcome-based performance measures for the Army tactical network. In addition, both DOD and Army officials indicated they are planning to develop a preferred end-to-end performance projection for the Army tactical communications network and intend to quantify the performance needed in terms of voice, data, and so forth, and by network tier, sector, and subnet. Officials plan to define levels of performance for benign and conflict environments and the waveforms and radios soldiers will need for each tier as well as their specific performance characteristics. Although this effort is in its early stages, this DOD oversight official stated that it is expected that the NIE will generate data on performance of the network as a whole, which could then be compared to the expected performance demand.

Separately, the Army is also beginning to prepare qualitative assessments of the progress the Army is making in filling capability gaps related to mission command essential capabilities. For example, ATEC has prepared an integrated network assessment after each NIE, which characterizes the level of capability achieved against the mission command essential capabilities. In addition, the Army has prepared a limited assessment of how capability set 13 will meet mission command essential capabilities.

Once the performance measures are in place and the Army evaluates the delivered capabilities against those measures, the Army will have the tools to evaluate the progress it is making and make any necessary adjustments to its investment strategy.

Opportunities for Additional Collaboration between Army and Test Community

The Army’s network strategy features a variety of different approaches to testing and evaluation to accommodate the rapid pace of technology change and to reduce the cost and time of acquisition. The Army has worked closely with the test community to plan, conduct, and evaluate the NIEs. Also, as mentioned earlier, the test community has taken a number of actions to reduce the costs of planning and executing the NIEs. At the same time, the test community has been meeting its responsibility to objectively report on the tests and the results. However, test results for several network systems at the NIEs that did not meet operational and other requirements will result in added time and expense to address identified issues. An inherent value of testing is pointing out key performance, reliability, and other issues that need to be addressed as soon and as economically as possible, but not after fielding. DOT&E has stated that the schedule-driven nature of the NIEs contribute to systems moving to testing before they have met certain criteria.

Tension between the acquisition and testing communities has been long- standing. In that regard, the Defense Acquisition Executive recently chartered an independent team to assess concerns that the test community’ approach to testing drives undue requirements, excessive cost, and added schedule into programs and results in a state of tension between program offices and the testing community.

One area the Defense Acquisition Executive assessment identified for improvement was the relationship and interaction among the testing, requirements, and program management communities. In that regard, the memorandum reporting the results called attention to four specific issues those communities need to address the need for closer coordination and cooperation among the requirements, acquisition, and testing communities; need for well-defined testable requirements; alignment of acquisition strategies and test plans; and need to manage the tension between the communities.

Concurrently, a systematic review of recent programs by DOT&E and DT&E examined the extent to which testing increases costs and delays programs. The results of both efforts indicated that testing and test requirements by themselves do not generally cause major program delays or increase costs. In addition, the Defense Acquisition Executive found no significant evidence that the testing community typically drives unplanned requirements.

Further, according to the DOT&E fiscal year 2012 annual report, three specific areas exist where increased test community interactions could result in improved test outcomes, which can result in systems with needed and useful combat capability being delivered to our forces more quickly. These include developing mission-oriented metrics to evaluate each system within the context within which it will operate; leveraging test and evaluation knowledge in setting requirements; and evaluating the multiple conditions in which the system is likely to be operated.

Additional opportunities exist for leadership of the Army and the test community to work together to further improve NIE execution and results. A good starting point would be for the Army to consider addressing the test community observations and recommendations from previous NIEs. Those included the schedule driven nature of NIEs, the lack of well- defined network requirements, and the lack of realistic battlefield maintenance and logistical support operations for SUTs during the NIEs. The Army is not required to and has not directly responded to the test community about its NIE observations and recommendations. Nevertheless, per internal control standards, managers are to, among other things, promptly evaluate findings from audits and other reviews, including those showing deficiencies and recommendations reported by auditors and others who evaluate agencies’ operations. In doing so, the Army may not only improve NIE execution and results but also reduce the tensions with the test community.

Conclusions

Within a sizable investment of an estimated $3 billion per year to modernize its tactical network, the Army is investing over $150 million per NIE to help ensure that those planned development and procurement investments result in the expeditious delivery of increased capabilities to the warfighter.

The main product of the NIEs is knowledge. The Army has not consistently recognized, accepted, and acted upon the knowledge gained from the NIEs. On the one hand, the Army’s fielding decisions to date seem driven by a pre-determined schedule rather than operational test results. Fielding individual systems that have done poorly during operational tests carries risk of less than optimal performance with the potential of costly fixes after fielding and increased operating and sustainment costs. Moreover, performance and reliability issues of individual systems could be magnified when these systems become part of an integrated network. On the other hand, even with a new strategy for procurement of emerging capabilities to fill capability gaps, the Army may still face an expectation gap with industry. The current constrained budget environment and the level of funding already allocated to ongoing network acquisition programs, may leave little funding to procure new networking technologies. Until it has clearly demonstrated the means to rapidly buy and field emerging capabilities and provided this information to industry, the Army may need to manage industry expectations of how many new networking systems it can buy and how rapidly.

The Army has implemented some lessons learned from planning and executing the NIEs. However, as part of a knowledge-based approach to its broader network modernization strategy, the Army should also be open to consideration of observations from all sources to improve process efficiency and achieve improved outcomes. We believe that the Army can and should collaborate more extensively with the test community on a variety of issues that could improve NIE outcomes. For example, as part of its responsibility to objectively conduct tests and report on their results, the test community has provided reports, observations, and recommendations before and following NIEs. To date, the Army has not directly responded to the test community’s observations and recommendations on the NIEs.

Recommendations for Executive Action

To improve outcomes for its entire network modernization strategy, we recommend that the Secretary of Defense direct the Secretary of the Army to take the following four actions:

Require that network systems from major defense acquisition programs obtain a positive Assessment of Operational Test Readiness (now called a Developmental Test and Evaluation Assessment) recommendation before being scheduled for operational testing during the NIE;

Correct network system performance and reliability issues identified during the NIEs before moving to buy and field these systems;

Provide results to industry on the Army’s actual experience in buying and fielding successfully demonstrated systems under evaluation and the length of time it has taken to date; and

Collaborate with all network stakeholder organizations to identify and correct issues that may result in improved network outcomes, including addressing the observations and recommendations of the test community related to the NIEs.

Agency Comments and Our Evaluation

DOD’s written response to this draft is reprinted in appendix II. DOD also provided technical comments that were incorporated as appropriate.

DOD partially concurred with our recommendations that the Army (1) require network systems obtain a positive Assessment of Operational Test Readiness (now called a Developmental Test and Evaluation Assessment) recommendation before being scheduled for operational testing during the NIE and (2) correct network system performance and reliability issues identified during the NIEs before moving to buy and field these systems. In both cases, DOD states that processes are already in place to address these issues and that the recommendations as written take flexibility away from the Department. We disagree. Our findings indicate that DOD is not using its current processes effectively to evaluate a system’s readiness to begin operational testing. While there may be instances where the Army uses operational testing to obtain feedback on system performance, DOD’s system development best practices dictate that a system should not proceed to operational testing until it has completed developmental testing and corrected any identified problems.

The NIEs are a good forum for the Army to generate knowledge on its tactical network. However, NIEs are a large investment and DOD and the Army should strive to optimize their return on that investment. Approving network systems for operational testing at the NIEs after having poor developmental test results may not be the best use of NIE resources because of the strong correlation between poor developmental test results and poor operational test results. Moreover, it is much more cost effective to address performance and reliability issues as early as possible in the system development cycle and well in advance of the production and fielding phases. As we note in the report, DOD and the Army have been pursuing a schedule-based strategy for network modernization rather than the preferred event-based strategy where participation in a test event occurs after a system has satisfied certain criteria.

DOD concurred with our recommendation that the Army provide results to industry on how many successfully demonstrated systems under evaluation have been procured to date and how long it has taken for the procurements. However, DOD did not offer specific steps it would take to provide this information or a proposed timeframe. Because of the importance of continued industry participation in the development of the Army network, we think that it is important for industry to have a clear picture of the Army’s success in rapidly buying and fielded emerging technologies. Finally, DOD concurred with our recommendation that the Army collaborate with all network stakeholder organizations to identify and correct issues that may result in improved network outcomes, including addressing the observations and recommendations of the test community related to the NIEs. DOD states that a collaborative environment with all stakeholders will assist in identifying and correcting issues and that the forum for doing so is the semiannual Network Synchronization Working Group. We agree that a collaborative environment is important in responding to previous test community observations and recommendations and would expect the Working Group to address these issues.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of the Army, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact Belva Martin at (202) 512-4841 or martinb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

Our objectives were to evaluate (1) the results of the Network Integration Evaluations (NIE) conducted to date and identify the extent to which the Army has procured and fielded proposed network solutions; and (2) Army actions and additional opportunities to enhance the NIE process. To address these objectives, we interviewed officials from the Army’s System of Systems Integration Directorate; the Deputy Chiefs of Staff, G-3/5/7 and G-8; the Army Brigade Modernization Command, and the Army Test and Evaluation Command. We met with representatives of Army Brigade Combat Teams preparing for deployment. We also interviewed officials from the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation; the Director, Operational Test and Evaluation; and the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. We visited the Lab Based Risk Reduction facility at Aberdeen Proving Ground, Maryland and the NIE test site at White Sands Missile Range, New Mexico to meet with soldiers and civilian officials conducting testing.

To examine the results of NIEs conducted to date, we attended Network Integration Evaluations and reviewed test reports from the Brigade Modernization Command, U.S. Army Test and Evaluation Command, the Director of Operational Test and Evaluation, and the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation. We reviewed briefing presentations for Army leadership that discuss test results and recommendations, and we toured lab facilities to understand how the Army is validating and selecting technologies for network evaluations. We reviewed Army programmatic and budget documentation to understand cost projections for testing and procuring network equipment under the new approach and we reviewed Army plans for resourcing this approach.

To identify actions and opportunities to enhance the NIE process, we interviewed Army officials to identify other networking challenges the Army is addressing concurrent with implementation of the agile process. We reviewed test results from both the Army and Department of Defense. We reviewed Army documentation identifying cost avoidance opportunities. We reviewed briefing information regarding lessons learned from activities related to the NIE, such as the screening and lab testing of candidate systems and soldier training. We spoke with officials at both Army and Department of Defense knowledgeable of lessons learned for the testing and fielding of new network capabilities.

We conducted this performance audit from September 2012 to August 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, William R. Graveline, Assistant Director; William C. Allbritton; Marcus C. Ferguson; Kristine Hassinger; Sean Seales; Robert S. Swierczek; and Paul Williams made key contributions to this report.