Army Workload and Performance System: Actions Needed to Complete Assessment of Unnecessary Overlap with Logistics Modernization Program
Released: 2014-02-28T00:00:00

Letter

Background

The Army began developing AWPS in 1996 at the direction of the House National Security Committee (now known as the House Armed Services Committee), and started fielding the system to its maintenance depots that year. AWPS was not intended to directly replace any single existing system; rather it was aimed at resolving deficiencies in the Army’s civilian manpower requirements determination process by providing analytically based workload forecasts to support workforce requirements. In 1998, the committee further directed the Army to develop a master plan for implementing AWPS. We subsequently reviewed the master plan and recommended, among other things, that the Army examine its existing program management and oversight structure as it continued to implement AWPS. The Department of Defense (DOD) concurred with the recommendations and cited corrective actions it would take, including having the Secretary of the Army assess the adequacy of existing program management and oversight structures responsible for system development and implementation. In June 2001, the Army sent an updated master plan to the House and Senate Committees on Armed Services.

Section 346 of the National Defense Authorization Act for Fiscal Year 2002 encouraged the sharing of data between AWPS and LMP (which had been in development since 1999) and identified several requirements related to AWPS. Among other things, the law required: the Army to continue to use AWPS as a standard Army-wide manpower system under the supervision and management of the Secretary of the Army; the Secretary of the Army to submit annual progress reports to Congress on the implementation of the AWPS master plan until the Secretary certifies to Congress that AWPS is fully implemented; and GAO to submit evaluations of each of the Army’s annual reports not later than 60 days after the Army has issued them to Congress. We evaluated the Army’s May 2002 progress report to Congress and recommended several actions to improve the quality of the Army’s annual progress reports and enhance the efficiency and effectiveness of AWPS. DOD concurred with our recommendations and stated that it would take steps to implement them. For example, the Army said it was in the final stages of updating the AWPS master plan and that it would provide it to Congress by November 2007. However, the Army ultimately did not submit an updated master plan. In our most recent report on AWPS in July 2011, we found that the Army was using AWPS to generate reports related to managing its workload and workforce issues but that the accuracy of those reports varied. We also found that the Army’s oversight of AWPS was fragmented among several organizations and that the Army had not been submitting annual reports on the implementation of the 2002 AWPS master plan. Accordingly, we recommended that an organization be identified for overall oversight of AWPS and that annual reports on AWPS implementation be submitted to Congress, as required by law. The Army concurred with these recommendations and stated that it would update the master plan and report to Congress by October 2011. We discuss later in this report the status of the Army’s efforts to address this recommendation.

AWPS has several modules that enable users to generate products that include the workload and resources report, which compares forecast workload against the available workforce; performance management and control tables, which assess the cost and schedule performance for production programs; and a report used by AMC leadership to monitor the status of efforts to repair and recapitalize equipment returning from overseas operations. To do so, AWPS pulls LMP data on labor costs as well as the personnel time required to complete repairs, among other things. Officials from Army G-4 (Maintenance) also use AWPS reports to assess whether the Army is maintaining a core depot-level maintenance and repair capability. As of October 2013, approximately 1,400 personnel had access to the system at 15 AMC industrial sites.

The Army spent approximately $1.4 billion on LMP through fiscal year 2012, and its total life cycle cost from fiscal years 2000 to 2026 is more than $4 billion. According to the AWPS program management office, the Army has spent over $90 million on AWPS from its inception in 1996 through the end of fiscal year 2013. Table 1 shows the Army’s reported actual and projected AWPS costs from fiscal year 2011—when we issued our last report on AWPS—through fiscal year 2018. The costs include government and contractor personnel expenses and hardware and software upgrades.

The Army Has Not Followed Certain Statutory and Regulatory Requirements for Implementing AWPS

The Army Has Not Submitted Annual Progress Reports to Congress on AWPS Implementation

The Army has not submitted the required annual progress reports to Congress on the implementation of the AWPS master plan since 2002. Section 346 of the National Defense Authorization Act for Fiscal Year 2002 requires the Secretary of the Army to submit annual progress reports to Congress on the implementation of the AWPS master plan until the Secretary certifies to Congress that AWPS is fully implemented. The Army submitted its first AWPS progress report to Congress in 2002 in the form of a revised version of the AWPS master plan. Although the Secretary of the Army has not yet certified to Congress that AWPS is fully implemented, the Army has not submitted any further annual reports. Because we recommended in July 2011 that the Army submit annual reports on the implementation of the AWPS master plan, as required by law, we are not making a new recommendation on this issue in this report.

The Army has not been submitting annual reports because, according to Army officials, it does not plan to submit progress reports until the revised master plan is completed. Army officials stated that AMC, in coordination with other Army offices, has been drafting an update to the AWPS master plan since July 2011—when GAO’s last report on AWPS was issued—in order to reflect changes regarding AWPS status and future use. However, it has not yet been completed and submitted to Congress, and officials were uncertain when that would occur. The AWPS master plan is still important, as AWPS has not yet been certified as fully implemented. According to officials, the Army has not completed the AWPS master plan due to oversight challenges arising from a series of internal reorganizations and multiple transfers of responsibility for AWPS within the Army. Another contributing factor, according to AMC officials, was the loss of personnel with knowledge of AWPS that has occurred since the AWPS master plan was last updated in 2002. In response to one of our July 2011 recommendations, the Army identified the Office of the Assistant Secretary of the Army (Manpower and Reserve Affairs) as the Army organization with overall oversight of AWPS. However, we did not find evidence that this office has had an active role in overseeing the completion of the AWPS master plan. Internal control standards identify the need for oversight, including responsibilities such as setting and monitoring objectives, as important. In the absence of oversight, completion of the AWPS master plan could be further delayed, resulting in continued lack of submission of annual progress reports on AWPS implementation to Congress.

The Army Has Not Validated AWPS as a Manpower Requirements Determination Tool

The Army has not validated AWPS as a manpower requirements determination tool as required by regulation. In October 2001, the Secretary of the Army designated AWPS as an Army-wide decision support, manpower determination, and management tool for applicable workload and workforce in the Army. According to Army regulation, USAMAA is responsible for reviewing and validating manpower requirements models. Army Regulation 570-4, which establishes policies and procedures for manpower management, states that USAMAA is responsible for reviewing and recommending approval of manpower requirements models developed by major commands. Army Regulation 71-32, which establishes policy for force development and documentation, states that the Assistant Secretary of the Army (Manpower and Reserve Affairs), to which USAMAA provides assistance, is responsible for validating manpower models. Such reviews are to occur at least every 3 years, or more often as needed, and are part of USAMAA’s responsibility to ensure consistent application of Army manpower policies. Army regulation also states that major commands— such as AMC—are responsible for submitting manpower models to USAMAA for approval, along with background information to support assumptions and equations in existing and proposed models. According to USAMAA officials, AWPS has not been validated as a manpower requirements determination tool in accordance with Army regulation, and therefore they cannot be assured that AWPS reports on workforce requirements are accurate.

USAMAA officials stated that they have previously asked AMC whether it would submit AWPS for review and validation, but AMC has not yet done so. USAMAA officials told us they contacted AMC in late 2011 or early 2012 regarding submitting AWPS for validation. At the time, AMC was not prepared to move forward with validation. However, officials from the AWPS program management office told us that in April 2012 they asked for assistance from USAMAA regarding submitting AWPS for validation, but that additional assistance had not yet been provided. Because USAMAA officials involved in these earlier contacts subsequently left USAMAA, we were unable to corroborate that USAMAA did not respond to AWPS program management officials’ requests for assistance. A senior official within the Office of the Assistant Secretary of the Army (Manpower and Reserve Affairs) told us that USAMAA had provided assistance to the AWPS program management office but that AWPS had not been formally submitted for review and validation.

USAMAA has published written guidance on how to develop and validate a manpower modeling tool. This guidance is designed for developing new tools and is not tailored for submission of an existing tool, such as AWPS, for validation. For example, the primary validation process identified in the guidance begins at the outset of the model-development process, which is not applicable for an existing tool. However, USAMAA officials provided us with an explanation of what would be necessary to validate AWPS as a manpower requirements determination tool and indicated that they work collaboratively with Army organizations to support the model-validation process. The process would include a briefing from AMC that demonstrates the AWPS manpower model. USAMAA officials could also ask AMC for further information on the model. USAMAA would then test the model to ensure the accuracy of what was presented by AMC. If any issues were to arise during testing, USAMAA would work with AMC to address them before validating the AWPS manpower model. Officials from the AWPS program management office stated that although USAMAA has not validated AWPS as a manpower requirements determination tool, AWPS has undergone prior reviews by other entities, to include an assessment of the data interface between AWPS and LMP. As of September 2013, the draft of the updated AWPS master plan states that the Army plans to submit AWPS to USAMAA for validation in the next 1 to 4 years, and AMC officials also acknowledged the need to have AWPS validated by USAMAA.

Although AWPS has not been validated, Army officials stated that AWPS reports continue to be used for workforce planning, including the development of budget estimates for future workload and workforce needs. For example, AMC industrial sites are using AWPS forecasting reports to make decisions on how to size their workforce to the sites’ projected workload. We previously reported that the accuracy of reports generated from AWPS varied. Through USAMAA validation of AWPS, the Army would have greater assurance that reports of these types, linking workload demand to workforce requirements, are accurate.

The Army Has Begun to Assess Unnecessary Overlap between AWPS and LMP, but Progress Has Been Limited

The Army has begun to assess whether unnecessary overlap exists between AWPS and LMP and has begun its assessment by initiating data-collection and analysis efforts. However, its overall progress— including the development of a business case analysis—has been limited, due primarily to the absence of senior-level leadership involvement and attention and the lack of a fully developed and documented approach for conducting the assessment, including an established milestone for completing it. GAO has found that overlap among government programs or activities can lead to unnecessary duplication and can result in unnecessary costs and less-efficient and less-effective services. Eliminating unnecessary overlap is particularly important given the current budgetary situation that the federal government and DOD face.

In December 2012, the AMC Commanding General and Deputy Commanding General directed that a plan be developed to “sunset,” or retire, AWPS and transition its functionality to another system, such as LMP. AMC officials stated that this task was assigned by AMC leadership as a result of feedback provided by industrial sites indicating that overlap existed between LMP and AWPS and questioning whether it was necessary for sites to continue to pay operating expenses for both AWPS and LMP. In response, a project team was established within AMC and started meeting in February 2013 to assess unnecessary overlap between AWPS and LMP. As part of this effort, the team has begun to evaluate the extent to which software functionality currently provided by AWPS can be provided by LMP—either through LMP’s existing functionality or through the expansion of functionality planned for LMP Increment 2. AMC officials stated that LMP Increment 2 will include a software module called Business Objects that will be able to produce reports similar to some of those produced by AWPS—such as charts outlining cost performance and schedule performance for projects at AMC industrial sites. Team members stated they will develop a business case analysis that will likely recommend one of three courses of action for providing AWPS functionality: retire AWPS completely and provide functionality with another system, such as LMP; provide functionality partially with AWPS and partially with another system, such as LMP; or continue to provide functionality solely with AWPS.

The project team has begun developing its assessment by initiating data- collection and analysis efforts. Specific actions to complete the assessment have been identified, such as surveying AWPS users, holding a design workshop to determine what AWPS functionality can be provided by LMP or another tool, and developing the business case analysis mentioned previously. The team has received a demonstration on the current functionality provided by AWPS and LMP and has distributed surveys to Army personnel with AWPS accounts to collect information on how they use the system. Among other things, the survey asked whether personnel used AWPS, for what purposes, and how often. The survey also asked whether they used another tool—such as LMP—to provide similar functionality. The team received survey responses from several AMC sites, including consolidated responses signed by site leadership (e.g., depot commander) and individual responses from AWPS account holders. At the time of our assessment, consolidated responses were provided by 7 of 15 AMC sites—two life cycle management commands, three maintenance depots, and two manufacturing arsenals.

Based on our review of the consolidated responses to the project team’s survey and our previous work, there is potential for some unnecessary overlap between AWPS and LMP, but further analysis is necessary. The survey lists several AWPS modules and asks users to report the time spent using each module, as well as the time spent using other tools to perform similar functions. All seven consolidated survey responses reported that AMC sites use some AWPS modules, but they also all reported using other tools—including LMP—to provide similar functionality. However, the project team has not assessed the reason these sites use multiple tools to perform similar functions and therefore it has not yet determined the extent of unnecessary overlap that may exist. For example, Anniston Army Depot and Rock Island Arsenal responded that they use reports available in LMP to assess cost performance and schedule performance for projects at their sites. Other sites responded that they use locally developed tools that extract data from LMP and produce reports similar to those in AWPS. For example, Red River Army Depot responded that it has developed a depot information visualization system that extracts data from LMP and links planned workload to workforce requirements. In addition, Tobyhanna Army Depot is using a tool in Microsoft Excel that extracts data from LMP and develops management reports similar to those created by AWPS. According to Tobyhanna’s survey response, this tool allows the depot to compare planned workload against its available workforce. Officials from the AWPS program management office stated that they plan to analyze the survey results and follow up with survey responders to obtain additional information. Furthermore, as previously noted, AMC initiated its assessment as a result of feedback provided by industrial sites indicating that overlap existed between LMP and AWPS.

Although the project team has made progress on the user survey, its progress in other areas of the assessment has been more limited. The team’s charter—which it began drafting in February 2013 to guide its operations—has not been completed. Team meetings have not been held regularly, and Army officials stated that attendance at these meetings is not always a high priority for team members. The AWPS program management office first provided a demonstration of AWPS functionality to the team in March 2013, but the LMP product office did not provide a similar demonstration of LMP functionality until December 2013. Milestones for some key actions—such as developing a business case for how to most cost-effectively provide AWPS functionality—have not been established, while milestones set for other actions have been missed. For example, the team originally planned to complete its analysis of survey responses from AWPS users by August 2013. However, as of December 2013, it had not completed this analysis.

Progress on the Army’s assessment of AWPS has been limited due to the absence of senior-level leadership attention and involvement. Best practices have shown that sustained leadership attention and involvement can help organizations make lasting changes and achieve positive results. In addition, these best practices indicate that a strong and stable team responsible for managing change is also essential to ensure that the effort receives the attention it needs to be sustained and successful. Having a strong, stable team for this assessment is particularly important because any changes made to AWPS and LMP could potentially affect numerous functions and activities across AMC. The project team is composed of representatives from various offices within AMC. The team leader works in the AMC G-1 (Personnel) office and has provided G-1 management with updates on the team’s progress. The other team members, however, do not work for G-1 and therefore do not officially report to the team leader or G-1 management; team members cited this leadership structure as a factor contributing to the team’s limited progress. Project team members stated that other than a June 2013 progress update to an AMC committee overseeing the implementation of LMP, no senior-level AMC manager or committee has requested updates on the team’s progress in assessing unnecessary overlap between AWPS and LMP and developing a plan to potentially retire AWPS.

Additionally, the team’s progress has been limited due to the lack of a fully developed and documented approach, including an established milestone for completing the assessment and recommending a course of action for providing AWPS functionality. When the task was originally assigned in December 2012, the AMC Commanding General and Deputy Commanding General did not establish a milestone for when the project team should complete the assessment, and the project team itself has not yet established a completion date. Standards for Internal Control call for proper documentation of evaluation processes. In addition, best practices have shown that a well-planned schedule—with milestones for completing activities—is a fundamental management tool that can help government programs use public funds effectively by specifying when work will be performed in the future and measuring program performance against an approved plan. The absence of senior-level AMC leadership involvement and attention, as well as the lack of a fully developed and documented approach that includes an established milestone for completion, increases the risk that the Army will not make sustained progress in its efforts to identify and eliminate unnecessary overlap between AWPS and LMP.

Conclusions

The Army has experienced delays in completing an update to the AWPS master plan. In the absence of strong oversight, completion of the master plan could be further delayed, resulting in continued lack of submission of statutorily required annual progress reports to Congress on the system’s implementation. In addition, AWPS was developed to address deficiencies in the Army’s manpower requirements determination process, but the Army has not validated AWPS as a manpower requirements determination tool in accordance with regulation and therefore lacks assurance that AWPS is producing valid results for workforce planning. Further, GAO has found that eliminating unnecessary overlap among government programs or activities can help prevent unnecessary duplication and help agencies provide more-efficient and more-effective services. Eliminating unnecessary overlap is particularly important given the current budgetary situation that the federal government and DOD face. The Army’s current effort to assess unnecessary overlap between AWPS and LMP could facilitate its ability to make a sound business decision on how to most cost-effectively provide AWPS functionality—which could include retiring AWPS and transitioning its functionality to LMP. Timely and effective actions to complete the assessment could improve the Army’s ability to support industrial operations in the most-efficient and cost-effective manner by minimizing unnecessary overlap between these two information systems. In the absence of increased leadership attention and involvement and without a fully developed and documented approach for this assessment, including an established milestone for completion, the Army could continue— unnecessarily—to spend millions of dollars annually on AWPS.

Recommendations for Executive Action

We are recommending that the Secretary of the Army take four actions.

To ensure completion of the update to the AWPS master plan, we recommend that the Secretary of the Army direct the Assistant Secretary of the Army (Manpower and Reserve Affairs) to strengthen oversight of the plan’s development, review, and approval.

To provide greater assurance of the accuracy of manpower requirements reports produced by AWPS for use at Army industrial sites, we recommend that the Secretary of the Army direct AMC—with assistance as needed from USAMAA—to submit AWPS to USAMAA for review and validation as a manpower requirements determination tool, in accordance with Army regulations.

To complete the Army’s assessment of unnecessary overlap between AWPS and LMP and to ensure that a sound business decision is made on how to most cost-effectively provide AWPS functionality, we recommend that the Secretary of the Army direct the Commanding General, AMC, to take the following two actions: Identify a specific senior-level AMC manager or committee to provide increased leadership involvement and attention of the project team’s efforts, to include ensuring that a strong and stable team exists for managing change.

Establish a fully developed and documented approach for the team’s assessment, including a milestone for completing it.

Agency Comments

We requested comments from the Army, but none were provided.

We are sending copies of this report to appropriate congressional committees, the Secretary of Defense, and the Secretary of the Army. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-5257 or merrittz@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I.

Appendix I: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Thomas Gosling (Assistant Director), Joanne Landesman, Brian Mazanec, Alexis Olson, Geoffrey Peck, and Sabrina Streagle made key contributions to this report.