DOD Personnel: Documentation of the Army's Civilian Workforce-Planning Model Needed to Enhance Credibility
Published: 2003-08-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Between fiscal years 1989 and 2002, the Department of Defense (DOD) reduced its civilian workforce by about 38 percent, with little attention to shaping or specifically sizing this workforce for the future. As a result, the civilian workforce is imbalanced in terms of the shape, skills, and experience needed by the department. DOD is taking steps to transform its civilian workforce. To assist with this transformation, the department is considering adopting an Army workforce-planning model, known as the Civilian Forecasting System (CIVFORS), which the Army uses to forecast its civilian workforce needs. Other federal agencies are also considering adopting this model. GAO was asked to review the adequacy of the steps the Army has taken to ensure the credibility of the model.
section_title: What GAO Found
paragraphs: The Army has taken adequate steps to ensure that the historical personnel data used in the model are sufficiently reliable and that the information technology structure adequately and appropriately supports the model. For example, the Army has established adequate control measures (e.g., edit checks, expert review, etc.) to ensure that the historical data that goes into the model are sufficiently reliable. Moreover, it has taken adequate steps to ensure that the information technology support structure (i.e., the software and hardware used to interface with and house the model) would enable continuity of operations, functionality, and system modification and operations. However, the Army has not demonstrated that it has taken adequate steps to ensure that the model's forecasting capability provides the basis for making accurate forecasts of the Army's civilian workforce. The Army's original certification of CIVFORS in 1987 was based on a formal documented verification and validation of the model structure that has not been formally updated since that time even though the Army has undertaken several model improvements. According to the Army's CIVFORS program manager, the Army has taken several steps, to include an independent review, peer reviews, and a comparison of forecasted data to actual data. However, documentation of these steps is incomplete and, therefore, does not provide adequate evidence to demonstrate the credibility of the forecast results. Without adequate documentation, the Army cannot show that it has taken sufficient steps to ensure the model's credibility in terms of its forecasting capability; consequently, there exists a risk that the forecasts it produces may be inaccurate or misleading. Furthermore, without documentation of CIVFORS's forecasting capability, it may be difficult for DOD and other federal organizations to accurately determine its suitability for their use.

section_title: Letter
paragraphs: Between fiscal years 1989 and 2002, the Department of Defense (DOD)  reduced its civilian workforce from 1,075,437 to 670,166—about a 38  percent reduction—with little attention to shaping or specifically sizing  this workforce for the future. As a result, the civilian workforce is  imbalanced in terms of the shape, skills, and experience needed by the  department. DOD plans to downsize its civilian workforce by an additional  55,000 through fiscal year 2007. In addition, in April 2003, DOD submitted a  proposal to Congress that would authorize DOD to establish a National  Security Personnel System to transform its current civilian personnel  system. DOD is also exploring the feasibility of placing hundreds of  thousands of civilians into essentially nonmilitary jobs that are currently  performed by uniformed personnel. To assist in determining its future  workforce, DOD will need reliable workforce planning tools, such as  workforce projection models. According to DOD guidance, a model used  to provide data for decision making should be accredited--that is, the  model should be officially certified as acceptable for use for a specific  purpose.
In a February 2003 testimony, the Chief of Staff of the Army stated that the  Army has begun to transform its civilian personnel system. To assist with  this transformation, the Army is using its workforce-planning model,  known as the Civilian Forecasting System (CIVFORS), which forecasts  future civilian workforce needs. The Army is working closely with the  Office of the Secretary of Defense and the Office of Personnel  Management to demonstrate the applicability of the model for use DOD- wide and in other federal agencies. According to Army guidance (Army  Regulation 5-11 and Department of the Army Pamphlet 5-11), verification  is one of the steps needed to ensure a model’s suitability to perform its  intended purpose. The verification process evaluates the extent to which a  model has been developed using sound and established software  engineering techniques, and it establishes whether the computer code  correctly performs the intended functions. Army guidance also states that  assessment of the correctness and forecasting capability is required.
In this report, we reviewed the adequacy of the steps the Army has taken  to ensure the credibility of the model. In March 2003, we briefed your staff  on our preliminary findings. To determine the Army’s efforts to ensure the  credibility of its model, we interviewed and obtained pertinent  documentation from the Army’s CIVFORS program manager. We also  reviewed DOD and Army guidance relevant to the management of Army  models and interviewed DOD officials to discuss their plans to adopt  CIVFORS. We conducted our review from September 2002 to June 2003.  More detailed information on our scope and methodology appears at the  end of this report.
subsections: section_title: Background
paragraphs: According to an Army Human Resource official, the Army uses the  workforce-planning model—CIVFORS—for human resources  management. CIVFORS is a collection of software programs that  anticipate future impacts on the workforce so that management can plan  for changes instead of reacting to them. The model is used to evaluate a  number of critical areas in civilian workforce planning, including  projected recruitment of personnel, impact of organizational realignments,  and changes in workforce trends (such as aging, retention, and projected  personnel shortfalls). It is a life-cycle modeling and projection tool that  models the most significant events that describe the life-cycle path of  personnel, which includes accessions, promotions, reassignments,  retirements, and voluntary and involuntary separations over a 7-year  period.
Verification and validation of models are important steps to building  credible models because they provide the foundation for the accreditation  process to ensure the suitability of the models for their intended purposes,  as stated in Army guidance, Management of Army Models and  Simulations. The verification process evaluates the extent to which a  model has been developed using sound and established software  engineering techniques, and it establishes whether the model’s computer  code correctly performs the intended functions. Model verification  includes data verification, model documentation, and testing of the  information technology structure that supports the model; model  verification is contained in such documents as the programmer’s manual,  installation’s manual, user’s guide, analyst’s manual, and trainer’s manual.  According to Army guidance, assessment of the correctness and  forecasting capability of the model is also required, and it should be  performed by a subject matter expert independent from the model  developer; however, the developer is expected to conduct in-house  verification and testing to assist in the overall model development process.  Validation is the process of determining the extent to which the model  adequately represents the real world.
section_title: Civilian Workforce- Planning Model’s Data Reliability and Information Technology Structure Are Adequate, but Forecasting Ability Not Fully Established
paragraphs: The Army has taken steps to ensure the reliability of the historical  personnel data used by the model and the adequacy of its information  technology structure used to support the model, but it has not provided  documentation that it has sufficiently tested and reviewed the most critical  aspect of the model—its forecasting capability and the appropriateness of  its assumptions. As a result, the forecasting credibility of the current  version of the model is not sufficiently validated or documented. Without  proper documentation of the abilities of the model, there is a risk that the  forecasts it produces may be inaccurate or misleading and the suitability  for use by other organizations may be difficult to determine.
subsections: section_title: Historical Personnel Data Reliability Is Adequate
paragraphs: The Army’s review of the historical personnel data used to provide  information for workforce planning was adequate to show that the data  are sufficiently reliable for use in the workforce model. Data regarding  personnel (such as date hired, education, age, grade level, and  occupational series) are taken from the Army’s Workforce Analysis  Support System (WASS). CIVFORS uses the most recent 5 years of  historical data to forecast the civilian workforce planning needs during the  next 7 years.
According to Army guidance, to ensure that data are sufficiently reliable  for use in the Army model, support documents should contain information  about the overall characteristics of the database. Furthermore, the  documents should show the intended range of appropriate uses for the  model as well as constraints on its use. They should also include concise  statements of the condition of the database for the purpose of indicating  its stability. The Army provided most, but not all, of the documents  referred to in Army guidance; we believe that the documents provided are  key ones and are adequate to show that WASS data are sufficiently reliable  for use in CIVFORS. In addition, the Army program manager for the  CIVFORS workforce-planning model stated that the workforce data are  checked by reviewing the arithmetic in the numerical algorithms to verify  that there is no unexplained change in the size of the civilian personnel  workforce contained in the database. Further, edit checks include  matching social security numbers for personnel from one time period to  another to account for actual personnel and personnel transactions  processed. In addition, CIVFORS has automated checks for inappropriate  numbers or characters. Such steps help to assure that the data contained  in WASS accurately and completely reflect critical personnel aspects and  transactions.
section_title: Information Technology Structure Is Adequate
paragraphs: The Army’s procedures for validating the information technology support  structure (the software and hardware used to interface with and house the  model) were also sufficient. For example, the Army (1) adequately  documented the information technology structure to allow for continuity  of operations, (2) tested its functionality, and (3) provided expertise for  system modification and operation. Procedures used by the Army include  documenting the model’s system description and hardware and software  requirements, providing system and user manuals, planning for  configuration management, and conducting functionality tests to help  ensure the system’s usability and operability over time and to demonstrate  the adequacy of the information technology structure to support use of the  workforce model.
section_title: Model’s Forecasting Ability Is Not Fully Established
paragraphs: The Army’s documentation cannot show that the forecasting ability of  CIVFORS has been adequately evaluated and, therefore, we cannot fully  assess the credibility of the model. According to Army guidance, validation  is the process of determining the extent to which a model adequately  represents the real world. According to the Army program manager, over a  7-year period, CIVFORS forecasts the anticipated impacts on the  workforce based on the most significant events in the life-cycle path of  personnel (to include accessions, promotions, reassignments, retirements,  voluntary separations, and involuntary separations). Army guidance states  that an independent, peer, and subject matter expert review of the model  should be conducted. The Army guidance also suggests generally accepted  methods, such as conducting a careful line-by-line examination of the  model design and computer code and algorithms. The Army’s program  manager said this had been done for the original certification of CIVFORS  in 1987. However, no formal document of the reviews has been prepared  in the years since, even though the Army has undertaken several model  improvements, such as (1) an expanded scope to include more dimensions  in the modeling process; (2) a more integrated, streamlined process that  involves fewer steps; and (3) greater flexibility, achieved by generalizing  the formulas and parameters.
In addition, there is insufficient documentation regarding tests performed,  since 1987, in which CIVFORS’s forecasts for prior years are compared  against equivalent historical data (called an “out of sample” test) to  measure the model’s forecasting capability. Such testing, which is one  method to validate a model’s forecasting capability, would involve using  the first 5 of the last 7 years of historical data to forecast the 2 subsequent  years. The forecasts for the last 2 years could then be compared to the  actual historical data. The Army, however, performed tests comparing  patterns of forecasts against historical data (called “in sample” tests),  showing that forecasts reflect the same patterns as the historical data used  to develop them for a sample of three Army major commands. However,  the draft document that was provided to us was inadequate to fully assess  the sampling used by the Army and the value of the tests.
Finally, the Army could not provide adequate documentation of an  independent or peer review of the model. The Army’s CIVFORS program  manager stated that the major commands served as peer reviewers by  conducting a comparison of their workforce data to WASS and CIVFORS  workforce data. We believe that such assessments by users provide  important information but do not constitute a peer review as defined in  Army guidance. Also, the results of these assessments were not available  for us to review. The program manager also stated that an independent  subject matter expert reviewed the functional design and the code in 1999,  but a formal report of the activities performed and the specific changes or  modifications implemented during the review were not produced.
Documentation has often not been a priority for several reasons.  According to the Army’s CIVFORS program manager, lack of  documentation is primarily due to limited funding, which was spent on  implementing changes to CIVFORS and WASS rather than on the  production of formal documents. Further, a shortage of staff (only one  staff person—the program manager) and loss of documents during the  attack on the Pentagon on September 11, 2001, also affected the amount of  documentation the Army could provide us. The program manager also  stated that some documentation was not needed because CIVFORS’s  design is predicated on proven methods in other Army active-duty, military  manpower forecasting models. In addition, the program manager stated  that the Army and contractors have primarily been adapting technology  (upgrading from mainframe to personal computer to Web-based) to  improve model functionality rather than creating new technology.  However, without proper documentation of the abilities of the model,  there exists a risk that the forecasts it produces may be inaccurate or  misleading. Consequently, decisions about future workforce requirements  may be questionable, and planning for the size, shape, and experience  level of the future workforce may not adequately meet the Army’s needs.
These issues may extend beyond the Army. In April 2002, DOD published a  strategic plan for civilian personnel, which includes a goal to obtain  management systems to support workforce planning. According to a DOD  official responsible for civilian workforce planning tools, components  within DOD have been requesting a modeling tool to assist them with  civilian workforce planning. As a result, DOD has decided to test the  Army’s civilian forecasting model. In October 2002, DOD purchased  hardware, installed modified software, and provided training to a small  number of personnel. Recently, DOD obtained a historical database of  civilian personnel data from the Defense Management Data Center and  provided the database to the contractor to load into the model. Two  agencies have volunteered to test the model: the Defense Logistics Agency  and the Washington Headquarters Service. DOD is working to develop a  test for these organizations using their own civilian personnel data to test  the model. At the end of the testing period, DOD will assess the model to  obtain a better understanding of its logic and determine whether or not it  should be implemented departmentwide.
section_title: Conclusions
paragraphs: As DOD continues to transform and downsize its civilian workforce, it is  imperative that the department properly shape and size the workforce.  One tool that could assist in this effort is CIVFORS—the Army’s workforce  planning model. However, proper documentation of the verification and  validation of CIVFORS is needed before expanding its use. The Army has  taken adequate steps to ensure that the historical personnel data used in  the model are sufficiently reliable and the information technology  structure appropriately supports the model; however, it has not fully  documented that it has taken adequate steps to demonstrate the credibility  of the model’s forecasting capability. Further, a model should be fully  scrutinized before each new application because a change in purpose,  passage of time, or input data may invalidate some aspects of the existing  model. Without sufficient documentation to demonstrate that adequate  steps have been taken to ensure the credibility of the model’s forecasting  capabilities, decisions about the Army’s future civilian workforce may be  based on questionable data and other potential users cannot determine  with certainty the model’s suitability for their use.
section_title: Recommendation for Executive Action
paragraphs: To assure the reliability of Army civilian workforce projections, as well as  the appropriateness of the model for use DOD-wide and by other federal  agencies, we recommend that the Secretary of Defense direct the  Secretary of the Army to appropriately document the Army’s forecasting  capability of the model.
section_title: Agency Comments and Our Evaluation
paragraphs: Although DOD stated, in written comments on a draft of this report, that it  did not concur with our recommendation, the Army is taking actions that,  in effect, implement it. DOD’s written comments are contained in  appendix I.
Regarding our recommendation that the Secretary of Defense direct the  Secretary of the Army to appropriately document the Army’s forecasting  capability of the model, DOD stated that the Army recognizes the need to  fully document its verification and validation efforts. Further, DOD stated  the staff of the Assistant Secretary of the Army, Manpower and Reserve  Affairs, has begun developing a verification and validation plan to enable  outside parties to assess the suitability and adaptability of the model for  their organizational use. This verification and validation process is  scheduled for completion in September 2003. However, during our review,  DOD did not provide information about the full scope of this verification  and validation effort. We believe that as the Army undertakes its  verification and validation effort, it should clearly document, as we  recommended, its assumptions, procedures, and the results so that future  users can replicate the tests to appropriately establish the model’s validity  for their purposes.
DOD also did not concur with our finding that the forecasting ability of the  model has not been fully established. DOD stated that the ultimate test of a  system is performance and that CIVFORS has been consistently generating  Army projections with high standards of accuracy. We did not  independently evaluate the model’s accuracy. As our report makes clear,  our basic point is that the model’s forecasting ability has not been  documented in accordance with Army guidance. We continue to believe  that without adequate documentation, the Army cannot show that it has  taken sufficient steps to ensure the model’s credibility in terms of its  forecasting capability.  DOD also provided technical comments, which we  incorporated where appropriate.
section_title: Scope and Methodology
paragraphs: We did not independently evaluate the model or the application of the  steps; rather, we reviewed the adequacy of the steps that the Army  program manager stated were taken to ensure the credibility of the model.  To determine the adequacy of the steps the Army has taken to ensure the  credibility of its civilian workforce-forecasting model, we discussed  CIVFORS with the Army’s CIVFORS program manager in the Army G-1  office, Civilian Personnel Policy Directorate, who has overall  responsibility for the workforce analysis and the forecasting system. In  addition, Army contractor officials who are responsible for providing  technical, analytic, and management support to operate, maintain, and  enhance the planning tool and model participated in several of our  discussions with the program manager. We reviewed the following  CIVFORS’s documents regarding the information technology support  structure: the Configuration Management Manual, the System’s  Specifications, the Design/Subsystem Documentation, the Operator’s  Manual, and the User’s Manual. In addition, we reviewed the 1987 and  draft 2002 test analysis report on the Civilian Forecasting System and  other documentation provided by the Army to obtain information on how  the model operates according to model assumptions. We also reviewed the  DOD Defense Modeling and Simulation Office guidance on verification and  validation of models, the Army regulation and pamphlet pertaining to the  management of Army models and simulations, and other literature  regarding model credibility. We also interviewed DOD officials in the  Civilian Personnel Management Service responsible for developing plans  to adopt the Army’s workforce forecasting model to discuss the status of  their efforts.
We conducted our review from September 2002 to June 2003 in  accordance with generally accepted government auditing standards.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, the Under Secretary of Defense for  Personnel and Readiness, and the Secretary of the Army. We will also  make copies available to others upon request. In addition, the report will  be available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-5559. Key contributors to this report are listed in appendix II.
section_title: Appendix I: Comments from the Department of Defense
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the name above, David Dornisch, Barbara Johnson, Barbara  Joyce, John Smale, Dale Wineholt, and Susan Woodward made significant  contributions to this report.