DOD Systems Modernization: Management of Integrated Military Human Capital Program Needs Additional Improvements
Published: 2005-02-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) has long-standing problems with its information technology (IT) systems supporting military personnel and pay. To address these problems, DOD initiated the Defense Integrated Military Human Resources System (DIMHRS) program, which is to provide a joint, integrated, standardized military personnel and pay system across all military components. In November 2004, DOD accepted the design for the first of three phases, DIMHRS (Personnel/Pay). GAO reviewed DOD's management of the requirements definition for the system as well as the program's management structure.
section_title: What GAO Found
paragraphs: DOD faces significant management challenges with DIMHRS, a major system acquisition program that is expected to lead to major changes in the processing of military personnel and pay. To its credit, DOD has begun taking steps to ensure that the requirements and the design for the first phase of the program are consistent with each other by tracing backward and forward between the detailed requirements and the system design, and it did obtain formal user acceptance of the DIMHRS (Personnel/Pay) high-level requirements. However, it has not obtained user acceptance of the detailed requirements. Furthermore, it has not ensured that the detailed requirements are complete and understandable. For example, requirements for the interfaces between DIMHRS (Personnel/Pay) and existing systems have not yet been fully defined because DOD has not yet determined how many legacy systems will be partially replaced and thus require modification. Furthermore, DOD is still determining whether the data requirements provided to the contractor for system design are complete. Finally, an estimated 77 percent of the detailed requirements are difficult to understand, based on GAO's review of a random sample of the requirements documentation. These challenges increase the risk that the delivered system capabilities will not fully meet the users' needs. Moreover, although DIMHRS (Personnel/Pay) is to be an integrated system, its development is not being governed by integrated tools and approaches, such as an integrated program management structure, enterprise architecture, and master schedule. Furthermore, while DOD is appropriately attempting to maximize the use of commercial, off-the shelf (COTS) products in building the new system, it has not adequately followed some important best practices associated with COTS-based system acquisitions. For example, DOD's program plan/schedule does not adequately recognize the needs of end-user organizations for the time and resources to integrate DIMHRS (Personnel/Pay) with their respective legacy systems and to prepare their workforces for the organizational changes that the system will introduce. DOD's requirements definition challenges and shortcomings in program governance can be attributed to a number of causes, including the program's overly schedule-driven approach and DOD's difficulty in overcoming its long-standing cultural resistance to departmentwide solutions. Unless these challenges are addressed, the risk is increased that the system will not provide expected capabilities and benefits on time and within budget. Given the limitations in some DOD components' ability to accurately pay military personnel, it is vital that these risks be addressed swiftly and effectively.

section_title: Letter
paragraphs: As we first reported in 1993, the Department of Defense (DOD) has had  serious problems with military personnel and pay systems, including  shortcomings in its ability to properly pay military personnel and to  monitor and track them to, from, and within their duty stations. We  recently reported that such problems continue today, particularly for  Army Reserve and National Guard troops serving in Iraq and Afghanistan.  These long-standing problems can be attributed, in part, to    hundreds of supporting information technology (IT) systems, many of  which perform the same tasks and store duplicate data; the need for manual data reconciliation, correction, and entry across these  nonintegrated systems; and the large number of data translations and system interfaces.
To address these problems, DOD initiated the Defense Integrated Military  Human Resources System (DIMHRS) program, which is to provide a joint,  integrated system that is standardized across all military components.  The system is to be Web-based and maximize the use of commercial,  off-the-shelf (COTS) software. DOD plans to acquire and deploy DIMHRS  in three phases:    DIMHRS (Personnel/Pay)—military personnel hiring, promotion,  retirement, etc., and pay;   DIMHRS (Manpower)—workforce planning, analysis, utilization, etc.; and    DIMHRS (Training).
DOD accepted the design of the first system phase—DIMHRS  (Personnel/Pay)—in November 2004 and is now proceeding with  development of the system. Deployment to the Army and the Defense  Finance and Accounting Service (DFAS) is to begin in the second quarter  of fiscal year 2006, followed by deployment to the Air Force, Navy, and  Marine Corps.
The first phase, DIMHRS (Personnel/Pay), is intended to focus particularly  on    providing joint-theater commanders with accurate and timely human    providing active service members, reservists, and National Guard  members with timely and accurate pay and benefits, including those  performing in theaters of operation or combat; and   providing an integrated military personnel and payroll system that uses  standard data definitions across all services and service components,  thereby reducing multiple data entries, system maintenance, pay  discrepancies, and reconciliations of personnel and pay information.
The system design for DIMHRS (Personnel/Pay) is based on requirements  defined to reflect a new, joint approach to processing military personnel  and pay; this new approach is expected to result in major changes to  current business processes.
Given the importance of DIMHRS (Personnel/Pay) to DOD’s ability to  manage military personnel and pay, we initiated a review, under the  authority of the Comptroller General, of the management of the DIMHRS  (Personnel/Pay) acquisition. Our objectives were to determine    whether DOD has effective processes in place for managing the definition  of the requirements for DIMHRS (Personnel/Pay) and   whether DOD has established an integrated program management  structure for DIMHRS (Personnel/Pay) and is following effective processes  for acquiring a system based on commercial software components.
On November 30, 2004, we briefed officials from DOD’s Joint  Requirements and Integration Office, the Navy Program Executive Office  (IT), and DOD’s Joint Program Management Office on the results of our  review. This report transmits the briefing. The full briefing, including our  scope and methodology, is reprinted as appendix I. We performed our  work from January through November 2004, in accordance with generally  accepted government auditing standards.
subsections: section_title: DOD’s Management of the DIMHRS (Personnel/Pay) Requirements Definition Has Recently Improved, but Key Aspects of Requirements Definition Remain a Challenge
paragraphs: DOD has not effectively managed important aspects of the requirements  for DIMHRS (Personnel/Pay) to ensure that they are complete, correct,  and unambiguous. Requirements are the foundation for designing,  developing, and testing a system. Incorrect or incomplete requirements  have been commonly identified as a cause of systems that do not meet  their cost, schedule, or performance goals. Disciplined processes and  controls for the definition and management of requirements are defined in  published models and guides, such as the Capability Maturity Models  developed by Carnegie Mellon University’s Software Engineering Institute  and standards developed by the Institute of Electrical and Electronics  Engineers.
DOD’s management of DIMHRS (Personnel/Pay) requirements had several  shortcomings:    First, DOD did not initially ensure that the system requirements and  system design were aligned. DOD required the contractor to base the  system design only on high-level (more general) requirements, providing  detailed requirements to the contractor for information only. However,  according to the program office, the system design should be based on the  detailed requirements, and following our inquiries, the program office  began tracing backward and forward between the detailed requirements  and the system design to ensure consistency. Among other things, DOD is  analyzing financial system standards to ensure that all applicable  standards are included in the requirements and design for DIMHRS  (Personnel/Pay). Without consistency between requirements and design,  the risk is increased that the developed and deployed system will not fully  satisfy financial system standards and users’ needs.
Second, DOD did not ensure that the detailed requirements include  important content and that they are clear and unambiguous. The  requirements for the interfaces between DIMHRS (Personnel/Pay) and  existing systems are not yet complete because DOD has not yet  determined the extent to which legacy systems will be replaced and thus  require modification in order to interact with the new system.  Furthermore, DOD is still determining whether the data requirements  provided to the contractor for system design are complete. Finally, about  77 percent of the detailed requirements are difficult to understand, based  on our review of a random sample of the requirements documentation.  Our review showed that this documentation did not consistently provide a  clear explanation of the relationships among the parts of each requirement  (business rules; information requirements; and references to regulations,  laws, standards, and so on) or adequately identify the sources of data  required for computations. If requirements are not complete and clear,  their implementation in the system is not likely to meet users’ needs.
Third, DOD has not obtained user acceptance of the detailed requirements.
As we have pointed out, when business process changes are planned,  users’ needs and expectations must be addressed, or users may not accept  change, which can jeopardize the effort. One way to ensure and  demonstrate user acceptance of requirements is to obtain sign-off on the  requirements by end-user representatives. However, although the DIMHRS  (Personnel/Pay) program obtained the user organizations’ formal  acceptance of the high-level requirements, the process used to define the  detailed requirements has not resulted in such acknowledgment of  agreement on the requirements. Program officials stated that gaining  formal agreement from some of the user organizations would delay the  program and be impractical because of end users’ reluctance to accept a  set of joint requirements that requires end users to make major changes in  their current ways of processing military personnel and pay actions. We  have previously observed this challenge and have stated that DOD’s  organizational structure and embedded culture work against efforts to  modernize business processes and implement corporate information  systems such as DIMHRS (Personnel/Pay) across component lines.  Nevertheless, not attempting to obtain agreement on DIMHRS  (Personnel/Pay) requirements increases the risk that users will not accept  and use the developed and deployed system, and that later system rework  will be required to make it function as intended DOD-wide and achieve  stated military human capital management outcomes.
According to DIMHRS (Personnel/Pay) officials, a number of actions have  been taken to reduce the risk that users will not accept the system,  including conducting numerous focus groups, workshops, demonstrations,  and presentations explaining how the DIMHRS (Personnel/Pay) software  product could address DOD’s existing personnel/pay problems.
However, DIMHRS (Personnel/Pay) officials stated that support for the  system by the services’ executives is mixed. For example, the officials said  that (1) Army executives are committed to implementing and using the  DIMHRS (Personnel/Pay) system because they believe it will address  many problems that the Army currently faces; (2) Air Force officials  generally support the system but say they do not yet know whether the  system will meet all their needs; and (3) Navy and Marine Corps  executives are not as supportive because they are not fully convinced that  DIMHRS (Personnel/Pay) will be an improvement over their existing  systems.
The shortcomings in DOD’s efforts to effectively manage DIMHRS  (Personnel/Pay) requirements are attributable to a number of causes,  including the program’s overly schedule-driven approach and the difficulty  of overcoming DOD’s long-standing cultural resistance to departmentwide  solutions. These shortcomings leave DOD without adequate assurance that  the requirements will accurately reflect the end users’ needs and that the  resulting system design is reflective of validated requirements that will  fully meet DOD’s needs.
section_title: DOD Does Not Have a Well-Integrated Management Structure for DIMHRS (Personnel/Pay) and Is Not Following All Relevant Supporting Acquisition Management Processes
paragraphs: DOD does not have a well-integrated structure for managing DIMHRS  (Personnel/Pay), which DOD has described as an integrated program,  and it is not following some key supporting processes for acquiring  COTS-based business systems.
Program responsibility, accountability, and authority are diffused. Leading  organizations ensure that programs are structured to ensure that a single  entity has clear authority, responsibility, and accountability for the  program. For DIMHRS (Personnel/Pay), these are spread among three key  stakeholder groups whose respective chains of command do not meet at  any point below the Secretary and Deputy Secretary of Defense levels.  Responsibility for requirements definition rests with a joint requirements  development office, which is accountable through one chain of command.  Responsibility for system acquisition rests with the program office, which  is accountable through another chain of command. Responsibility for  preparing for transition to the new system rests with the end-user  organizations—11 major DOD components reporting through five different  chains of command. This is consistent with our earlier observation that  DOD’s organizational structure and embedded culture have not adequately  accommodated an integrated, departmentwide approach to joint systems.  Without a DOD-wide integrated governance structure for a joint,  integrated program like DIMHRS (Personnel/Pay), the risk is increased  that the program will not produce an integrated set of outcomes.
The system has not been defined and designed according to a DOD-wide  integrated enterprise architecture. In accordance with the National  Defense Authorization Act for Fiscal Year 2003, DOD has been developing  a departmentwide Business Enterprise Architecture (BEA), and it has  been reviewing some programs, such as DIMHRS (Personnel/Pay) with  proposed obligations of funds greater than $1 million, for consistency with  the BEA. In April 2003, the DOD Comptroller certified DIMHRS  (Personnel/Pay) to be consistent with the BEA on the basis of the program  manager’s commitment that the yet-to-be-developed system would be  designed to be consistent with the yet-to-be-developed architecture. To  follow through on this commitment, DOD included a requirement in the  DIMHRS (Personnel/Pay) contract that the systems specification be  compatible with the emerging BEA.
DIMHRS (Personnel/Pay) officials recognize that the April 2003  architectural certification is preliminary and stated that DIMHRS  (Personnel/Pay) will undergo another certification before the system  deployment decision. By that time, however, lengthy and costly design and  development work will have been completed. The real value in having and  using an architecture is knowing during system definition, design, and  development what the larger blueprint for the enterprise is, so that these  can be guided and constrained by this frame of reference. Aligning to the  architecture after the system is designed would require expensive system  rework to address any inconsistencies with the architecture.
Program stakeholders’ activities have not been managed according to a  DIMHRS (Personnel/Pay)-integrated master plan/schedule. An effective  master plan/schedule should allow for the proper scheduling and  sequencing of activities and tasks, allocation of resources, preparation of  budgets, assignment of personnel, and criteria for measuring progress.  However, the DIMHRS (Personnel/Pay) program plan/schedule is based on  the contractor’s and program office’s activities and does not include all the  activities that end-user organizations must perform to prepare for DIMHRS  (Personnel/Pay), such as the redesign of legacy systems and interfaces,  business process reengineering, and workforce change management.  Without a true master plan/schedule of activities that includes all DOD  program stakeholders, the risk increases that key and dependent events,  activities, and tasks will not be performed as needed, which in turn  increases the risk of schedule slippage and program goal shortfalls.
Some, but not all, best practices associated with acquiring COTS-based  business systems are being followed. An example of a best practice that  DOD is following is to discourage the modification of commercial software  components without thorough justification; DOD’s contract includes  award fees that give the contractor incentives to, among other things,  minimize the customization of the COTS software. An example of a best  practice that DOD is not following is to ensure that plans and schedules  explicitly provide for preparing users for the new business processes  associated with the commercial components. DOD does not have an  integrated program plan/schedule that provides for end-user organization  activities that are associated with preparing users for the changes that the  system will introduce. Because it is not following all best practices  associated with acquiring COTS-based systems, DOD is increasing the risk  that DIMHRS (Personnel/Pay) will not be successfully implemented and  effectively used.
DOD’s efforts to employ an integrated program management approach  have not been effective for a number of reasons, including DOD’s long- standing cultural resistance to departmentwide solutions. Without an  integrated approach and effective processes for managing a program that  is intended to be an integrated solution that maximizes the use of  commercially available software products, DOD increases the risk that the  program will not meet cost, schedule, capability, and outcome goals.
section_title: Conclusions
paragraphs: The importance of DIMHRS (Personnel/Pay) to DOD’s ability to manage  military personnel and pay services demands that the department employ  effective processes and governance structures in defining, designing,  developing, and deploying the system to maximize its chances of success.  For DIMHRS (Personnel/Pay), however, DOD did not initially perform  important requirements-development steps, and the detailed system  requirements are missing important content. DOD has begun to remedy  these omissions by taking actions such as tracing among requirements  documents and system design documents to ensure alignment, but user  organizations’ acceptance of requirements has not occurred. Moreover,  although DIMHRS (Personnel/Pay) is to be an integrated system, it is not  being governed by integrated tools and approaches, such as an integrated  program management structure, integrated DOD business enterprise  architecture, and an integrated master plan/schedule.
Furthermore, while DOD is appropriately attempting to maximize the use  of COTS products in building DIMHRS (Personnel/Pay) and is following  some best practices for developing COTS-based systems, others are not  being followed.
The absence of the full complement of effective processes and structures  related to each of these areas can be attributed to a number of causes,  including the program’s overly schedule-driven approach and the difficulty  of overcoming DOD’s long-standing cultural resistance to departmentwide  solutions. Effectively addressing these shortcomings is essential because  they introduce unnecessary risks that reduce the chances of  accomplishing DIMHRS (Personnel/Pay) goals on time and within budget.
It is critical that DOD carefully consider the risks caused by each of these  areas of concern and that it appropriately strengthen its management  processes, structures, and plans to effectively minimize these risks. To do  less undermines the chances of timely and successful completion of the  program.
section_title: Recommendations for Executive Action
paragraphs: To assist DOD in strengthening its program management processes,  structures, and plans and thereby increase its chances of successfully  delivering DIMHRS (Personnel/Pay), we recommend that you direct the  Assistant Secretary (Networks and Information Integration), the Under  Secretary (Personnel and Readiness), and the Under Secretary  (Comptroller), in collaboration with the leadership of the military services  and DFAS, to take the following six actions to jointly ensure an integrated,  coordinated, and risk-based approach to all DIMHRS (Personnel/Pay)  definition, design, development, and deployment activities. At a minimum,  this should include    ensuring that joint system requirements are complete and correct, and that  they are acceptable to user organizations;   establishing a DOD-wide integrated governance structure for DIMHRS  (Personnel/Pay) (1) that vests an executive-level organization or entity  representing the interests of all program stakeholders—including the Joint  Requirements and Integration Office, the Joint Program Management  Office, the services, and DFAS—with responsibility, accountability, and  authority for the entire DIMHRS (Personnel/Pay) program and (2) that  ensures that all stakeholder interests and positions are appropriately  heard and considered during program reviews and before key program  decisions;    ensuring that the degree of consistency between DIMHRS (Personnel/Pay)  and the evolving DOD-wide business enterprise architecture is  continuously analyzed and that material inconsistencies between the two,  both potential and actual, are disclosed at all program reviews and  decision points and in program budget submissions, along with any  associated system risks and steps to mitigate these risks;   developing and implementing a DOD-wide, integrated master  plan/schedule of activities that extends to all DOD program stakeholders;    ensuring that all relevant acquisition management best practices  associated with COTS-based systems are appropriately followed; and    ensuring that an event-driven, risk-based approach that adequately  considers factors other than the contract schedule continues to be used in  managing DIMHRS (Personnel/Pay).
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report (reprinted in app. II), the  Under Secretary of Defense for Personnel and Readiness stated that DOD  largely agrees with the thrust of our recommendations, and that it is  already following, to the extent practicable, the kind of acquisition best  practices embodied in them. The department also made two overall  comments about the report and provided a number of detailed comments  pertaining to five of our six recommendations.
The first overall comment was that our espousal of certain system  acquisition management best practices resulted in incongruity among our  recommendations. In particular, DOD indicated that our recognition that  DOD is appropriately limiting modification of COTS products (a best  practice) is incongruous with our recommendation that requirements be  acceptable to user organizations (another best practice). It further stated  that if it acted on all comments that it received on requirements from all  sources, as it suggested we were recommending, then this would result in  excessive modification to the COTS product. We do not agree with DOD’s  points; we suggest that a careful reading of our recommendations would  show that the department has not correctly interpreted and characterized  those recommendations that pertain to this overall comment. Specifically,  our report does not recommend that DOD act on all comments obtained  from all sources, regardless of the impact and consequences of doing so.  Rather, the report contains complementary recommendations for ensuring  that the system requirements are acceptable to user organizations and  discouraging changes to the COTS product unless the life-cycle costs and  benefits justify making them. In short, our recommendations concerning  system requirements are intended to provide DOD with the principles and  rules that it should apply in executing a requirements-acceptance process  that permits all stakeholder interests and positions to be heard,  considered, and resolved in the context of what makes economic sense.  While DOD’s comments note that a process was followed to screen out  user inputs that, for example, necessitated changes to the COTS product,  this process did not provide for the effective resolution of such inputs, as  shown in our report by certain user organizations’ comments: specifically,  that their involvement in defining detailed requirements was limited, that  their comments on these requirements were not fully resolved, and that  they were not willing to sign off on the requirements as sufficient to meet  their needs. This lack of resolution is important because not attempting to  obtain some level of stakeholder acceptance of requirements increases the  risk that the system will not adequately meet users’ needs, that users will  not adopt the system, and that later system rework will be required to  rectify this situation.
The second overall comment was that the department was already  employing acquisition management best practices, to the extent  practicable, and that the management process for the program is  innovative and groundbreaking for DOD, going far beyond what is required  by the department’s regulations. For example, the department commented  that the system-requirements documentation far exceeds that which has  been available for any other system effort. We do not dispute DOD’s  comment about efforts on this system relative to other system  acquisitions, because our review’s objectives and approach did not extend  to comparing DIMHRS (Personnel/Pay) with other DOD acquisitions.  However, our review did address DOD’s use of key acquisition  management best practices on DIMHRS (Personnel/Pay), and in this  regard we support the department’s recognition of the importance of these  practices. In our report, we have provided a balanced message by  recognizing instances where best practices were being followed, such as  when DOD began tracing detailed system requirements to the system  design following inquiries that we made during the course of our review.  However, we do not agree that at the time we concluded our work DOD  was following all relevant and practicable best practices; examples of  these practices are cited in our report and discussed below in our  response to DOD’s detailed comments on our individual  recommendations.
In its comments specific to our six recommendations, the department  agreed without further comment with one recommendation (to develop  and implement a DOD-wide, integrated master plan/schedule of activities  that extends to all DOD program stakeholders). In addition, it either  partially agreed or partially disagreed with our other five  recommendations, and it provided detailed comments on each. Generally,  DOD’s areas of disagreement relate to its view that it is already performing  the activities that we recommend.
DOD partially concurred with our recommendation to ensure that joint  system requirements are complete and correct and acceptable to user  organizations. In this regard, DOD stated that it has already taken great  pains to ensure that the requirements are complete and correct, although  its comments stated that this assurance has occurred “to the extent that  any documentation  this massive can be correct.” It also  stated that the requirements are fully traceable to the system design, and  that the high-level requirements were validated in accordance with DOD  regulations. It added that it has taken various steps to gain users’  acceptance of the system, including a change management process,  briefings, and prototype demonstrations.
We do not disagree that DOD has taken important steps to meet the goals  of requirements completeness and correctness. Likewise, we do not  disagree that since receiving our draft report for comment, the department  might have completed the important requirements-to-design traceability  steps that it began in response to our inquiries, which we describe in our  report. However, DOD’s comments contain no evidence to show that it has  addressed the limitations in the requirements’ completeness and  correctness that we cite in the report, such as those relating to the  interface and data requirements, and they do not address the  understandability issues we found relative to 77 percent of the detailed  requirements. Moreover, DOD stated in its comments that its latest  program review revealed 606 business process comments and 17 interface  comments that it deemed noncritical, although it noted that they were still  being analyzed.
We also do not disagree that DOD has taken steps to gain user acceptance  of the system. However, they did not gain acceptance of the detailed  requirements that the system is to be designed to meet, which is the focus  of our recommendation. As we point out in the report, not attempting to  obtain agreement on the detailed requirements increases the risk that  users will not adopt the system as developed and deployed, and that later  system rework will be needed to address this.
DOD partially concurred with our recommendation that it continuously  analyze the degree of consistency between DIMHRS (Personnel/Pay) and  the evolving DOD-wide BEA so that the risks of material inconsistencies  are understood and addressed. In doing so, DOD stated that the DIMHRS  (Personnel/Pay) requirements comprise the military personnel and pay  portion of the architecture and that as one of the first major systems  developed using all the principles of this architecture, DIMHRS  (Personnel/Pay) is and will remain fully consistent with it. We do not agree  with DOD’s comments that the system is consistent with the BEA. As we  state in our report, DOD could not provide us with documented, verifiable  analysis demonstrating this consistency and forming the basis for the DOD  Comptroller’s April 2003 certification of this consistency. Rather, we were  told that this certification was based on the DIMHRS (Personnel/Pay)  program manager’s stated commitment to be consistent at some future  point. However, as we note in our report, the real value of an architecture  is that it provides the necessary context for guiding and constraining  system investments in a way that promotes interoperability and minimizes  overlap and duplication. Without it, expensive system rework is likely to  be needed to achieve these outcomes. As we also note in our report, the  absence of verifiable analysis of DIMHRS (Personnel/Pay) architectural  compliance was in part due to the state of the BEA, which we have  reported as not being well-defined and missing important content.  Recognizing this, as well as the pressing need for DIMHRS’s  (Personnel/Pay) promised capabilities, our recommendation calls for  ongoing analysis of DIMHRS (Personnel/Pay) and the BEA to understand  the risks of designing and developing the system outside the context of a  well-defined architecture.
DOD partially disagreed with our recommendation that it establish a  DOD-wide governance structure in which responsibility, accountability,  and authority for the entire program are vested in an executive-level  organization or entity representing the interests of all program  stakeholders. In doing so, the department described the roles,  responsibilities, and authorities for various program stakeholders;  however, it did not explain its reason for not agreeing with the  recommendation, and only one of its comments bears relevance to our  recommendation. Specifically, it commented that the Under Secretary of  Defense (Personnel and Readiness) has full responsibility and  accountability for the program. We do not agree. As we state in our report,  DIMHRS (Personnel/Pay) is a DOD-wide program involving three distinct  stakeholder groups whose respective chains of command do not meet at  any point below the Secretary and Deputy Secretary of Defense levels.  Thus, we concluded that responsibility, accountability, and authority for  the program are diffused, with responsibility for developing functional  requirements resting with the Joint Requirements and Integration Office,  responsibility for system acquisition resting with the Joint Program  Management Office, and responsibility for preparing for the transition to  DIMHRS (Personnel/Pay) resting with 11 major end-user organizations.  Under this structure, only the Joint Requirements and Integration Office is  accountable to the Under Secretary, and the two distinct other stakeholder  groups are not accountable to the Under Secretary.
This means, as we state in the report, that no single DOD entity is  positioned to exercise continuous leadership and direction over the  entire program.
The department also partially disagreed with our recommendation to  follow all relevant acquisition management best practices associated with  COTS-based systems. According to DOD’s comments, all of these best  practices are currently being followed, including the three that we cite in  our report as not being followed: (1) ensuring that plans explicitly provide  for preparing users for the impact that the business processes embedded  in the commercial components will have on their respective roles and  responsibilities, (2) proactively managing the introduction and adoption of  changes to how users will be expected to use the system to execute their  jobs, and (3) ensuring that project plans explicitly provide for the  necessary time and resources for integrating commercial components with  legacy systems. In this regard, the department stated that the DIMHRS  (Personnel/Pay) program had documented every change in current  practices and policies that will be required for the military services, as well  as future practices and policies, and that these were fully vetted through  the functional user community. It also described a number of activities  that it has undertaken to prepare and train users in the COTS product and  other aspects of DIMHRS (Personnel/Pay).
We do not dispute whether DOD has performed activities intended to  facilitate the implementation of DIMHRS (Personnel/Pay). However, the  best practices that we identified as not being followed, which form the  basis of our recommendation, are focused on effectively planning for the  full complement of activities that are needed to prepare an organization  for the institutional and individual changes that COTS-based system  solutions introduce. Such planning is intended to ensure, among other  things, that key change management activities, including the dependencies  among these activities, are defined and agreed to by stakeholders,  including ensuring that adequate resources and realistic time frames are  established to accomplish them. In this regard, DOD agreed in its  comments that it does not have an integrated master plan/schedule for the  program, which is an essential tool for capturing the results of the  proactive change management planning that the best practices and our  recommendation advocate. Both published research and our experience in  evaluating the acquisition and implementation of COTS-based system  solutions show that the absence of well-planned, proactive organizational  and individual change management efforts can cause these system efforts  to fail.
The department partially disagreed with our last recommendation to adopt  a more event-driven, risk-based approach to managing DIMHRS  (Personnel/Pay) that adequately considers factors other than the contract  schedule, stating that it is currently using an event-driven, risk-based  approach and revising the schedule when necessary. We support DOD’s  comment, as it indicates that DOD has decided to begin following such an  approach. However, during the course of our work this was not the case.  For example, we observed at that time that the DIMHRS (Personnel/Pay)  program intended to accelerate its deployment schedule to meet an  externally imposed deadline and that it was not until we raised concerns  about the associated risks of doing so, as well as the absence of effective  strategies to mitigate these risks, in an earlier draft of the briefing included  in this report, that the department changed its plans. Also during the  course of our work, we observed that program activities were truncated or  performed concurrently in order to meet established deadlines. For  example, as we describe in our report, data requirements (which are  derived from higher-level information needs) were provided to the  contractor before information needs were fully defined because the  contractor needed these data requirements to complete the system design  on schedule. It was this kind of focus on schedule that led to our  recommendation to adopt a more event-driven, risk-based approach.  However, in light of DOD’s comment that it intends to do so, we have  slightly modified our recommendation to recognize this decision.
All our recommendations are aimed at reducing the risk of failure on this  important program, which we and DOD agree is critical to the  department’s ability to effectively manage military personnel and pay.  Furthermore, DOD’s comments show that it agrees with us on the  importance of taking an approach to the program that is based on the  kinds of management processes and structures that we recommend, and  the department appears committed to following such an approach.  Following our recommendations will help the department to do so and  thereby avoid unnecessary risks. As we state in the report, careful  consideration of the areas of concern that we raise is critical to improving  the chances of timely and successful completion of the program.
We are sending copies of this report to the House and Senate Armed  Services and Appropriations Committees; the House Committee on  Government Reform; the Senate Committee on Governmental Affairs; and  the Director, Office of Management and Budget. In addition, the report will  be available at no charge on the GAO Web site at http://www.gao.gov.
Should you or your offices have any questions on the matters discussed in  this report, please contact Randolph Hite at (202) 512-3439 or Gregory C.  Wilshusen at (202) 512-3317; we can also be reached by e-mail at  hiter@gao.gov or wilshuseng@gao.gov. Other contacts and key  contributors to this report are listed in appendix III.
section_title: Appendix I: Briefing to Department of Defense Officials, November 30, 2004
subsections: section_title: DOD Systems Modernization: Management of Integrated Military Human Capital Program Needs Additional Improvements
subsections: section_title: Briefing to Department of Defense Officials November 30, 2004
section_title: Attachment I: Objectives, Scope, and Methodology
section_title: These problems can be traced, in part, to such causes as
paragraphs: hundreds of supporting information technology (IT) systems, many of which  perform the same tasks and store duplicate data;  the need for manual data reconciliation, correction, and entry across these   the large number of data translations and system interfaces.
GAO, Financial Management: Defense’s System for Army Military Payroll Is Unreliable, GAO/AIMD-93-32  (Washington, D.C.: Sept. 30, 1993).
GAO, Military Pay: Army Reserve Soldiers Mobilized to Active Duty Experienced Significant Pay Problems,  GAO-04-911 (Washington, D.C.: Aug. 20, 2004), and Military Pay: Army National Guard Personnel  Mobilized to Active Duty Experienced Significant Pay Problems, GAO-04-89 (Washington, D.C.: Nov. 13,  2003).
section_title: The Defense Integrated Military Human Resources System (DIMHRS (Personnel/Pay)) is intended to address these problems, with particular focus on
paragraphs: providing joint-theater commanders with accurate and timely human capital   providing active service members, reservists, and National Guard members with  timely and accurate pay and benefits, especially when they are performing in  theaters of operation or combat; and  providing an integrated military personnel and payroll system that uses standard  data definitions across all services and service components, thereby reducing  multiple data entries, system maintenance, pay discrepancies, and  reconciliations of personnel and pay information.
Among other things, the new system is also intended to support DOD’s efforts to  produce accurate and complete financial statements.
DOD plans to acquire and deploy DIMHRS in three phases:   DIMHRS (Personnel/Pay)—military personnel hiring, promotion, retirement, etc.,   DIMHRS (Manpower)—workforce planning, analysis, utilization, etc.; and  DIMHRS (Training).
DOD accepted the design of the first system phase—DIMHRS (Personnel/Pay)—in  November 2004 and is now proceeding with development of this system phase.  Deployment to the Army and the Defense Finance and Accounting Service (DFAS)  is to begin in the second quarter of fiscal year 2006, followed by deployment to the  Air Force, Navy, and Marine Corps.
section_title: In view of the significance of the DIMHRS program to DOD’s ability to manage military personnel and pay services, we initiated a review under the authority of the Comptroller General. Our objectives were to determine
paragraphs: 1. whether DOD has effective processes in place for managing the definition  of the requirements for DIMHRS (Personnel/Pay) and  2. whether DOD has established an integrated program management  structure for DIMHRS (Personnel/Pay) and is following effective processes  for acquiring a system based on commercial software components.
To accomplish our objectives, we interviewed officials from relevant organizations,  analyzed program management documentation and activities, and reviewed  relevant DOD analyses. Further details on our scope and methodology are given in  attachment I to this appendix.
Our work was performed from January through November 2004, in accordance  with generally accepted government auditing standards.
Results in Brief: Objective 1 Requirements Management DOD’s management of the DIMHRS (Personnel/Pay) requirements definition has  recently improved, but key aspects of requirements definition remain a challenge.  In particular, DOD has begun taking steps to ensure that the system requirements  and the system design are consistent with each other. However, DOD  has not ensured that the detailed requirements are complete and   has not obtained user acceptance of the detailed requirements.
The requirements definition challenges are attributable to a number of causes  including the program’s overly schedule-driven approach and the difficulty of  overcoming DOD’s long-standing cultural resistance to departmentwide solutions.
These challenges increase the risk that the delivered system’s capabilities will not  fully meet DOD’s needs.
DOD does not have a well-integrated management structure for DIMHRS  (Personnel/Pay) and is not following all relevant supporting acquisition  management processes. In particular,  program responsibility, accountability, and authority are diffused;   the system has not been defined and designed according to a DOD-wide   program stakeholders’ activities have not been managed according to a master  plan/schedule that integrates all stakeholder activities; and  the program is following some, but not all, best practices associated with  acquiring business systems based on commercially available software.
Without an integrated approach and effective processes for managing a program  that is intended to be an integrated solution, DOD has increased the risk that the  program will not meet cost, schedule, capability, and outcome goals.
Results in Brief: Recommendations To assist DOD in effectively managing DIMHRS (Personnel/Pay), we are making  six recommendations to the Secretary of Defense aimed at ensuring that DOD  follows an integrated, coordinated, and risk-based program approach and thereby  increases its chances of successfully delivering DIMHRS (Personnel/Pay).
section_title: Program Chronology
section_title: May 1999. DOD’s Institute for Defense Analysis analyzed alternative systems solutions, reaffirmed the decision to base DIMHRS (Personnel/Pay) on commercial, off-the-shelf (COTS) software, and made a number of suggestions, including the need to favor the use of COTS “out of the box” to hold down costs.
subsections: section_title: At that time, this executive’s title was Assistant Secretary of Defense for Command, Control, Communications, and Intelligence.
section_title: May 2003. Assistant Secretary of Defense (NII) approved the program to proceed into the system development and demonstration phase (i.e., software development, integration, testing, etc.).
section_title: November 2004. DOD accepted the contractor’s design of DIMHRS (Personnel/Pay) and authorized the contractor to proceed with development of the system.
section_title: DIMHRS (Personnel/Pay) Description
section_title: According to DOD officials, DIMHRS (Personnel/Pay) will be the largest personnel system in the world in terms of the number of people it serves and transactions it processes.
section_title: Roles and Responsibilities of DIMHRS (Personnel/Pay) Stakeholders
paragraphs: Roles and responsibilities  Developing DOD integrated user requirements; providing expertise related to  personnel/pay functional area.   Acquiring the system, including managing the system development  contractor, accepting the design; and testing and deploying the system.  Assisting JR&IO in developing integrated requirements; assisting JPMO in  addressing technical issues; taking other actions necessary for transition.  Assisting JR&IO and JPMO in developing DIMHRS (Personnel/Pay);  managing transition activities for own chain of command, including modifying  existing systems and interfaces.
Under Sec. of Defense (P&R)  Overseeing personnel/pay functional area; resolving issues that cannot be  resolved by the Executive Steering Committee.
Executive Steering Committee  Monitoring the program, resolving issues, and advising the Under Secretary  (this is a stakeholder executive-level committee).  Monitoring the program and resolving issues (this is a group of user  representatives).  Designated Milestone Decision Authority.  Establishing acquisition policies and procedures in accordance with DOD  directives and guidelines and for chartering the Program Executive Office,  Information Technology (PEO-IT).  Providing programmatic and technical direction to JPMO.
section_title: Costs
section_title: DOD does not currently have a total life-cycle cost estimate for DIMHRS (Personnel/Pay). According to JPMO officials, JPMO’s costs through fiscal year 2003 and projected through fiscal year 2009 are about $601 million:
paragraphs: $244.7 million obligated during fiscal years 1998 through 2003 and   $356.6 million required for fiscal years 2004 through 2009.
However, JR&IO and JPMO officials stated that these amounts do not include user  organization costs; JPMO originally estimated these costs to be about $350 million,  but it is now reevaluating these and other cost estimates as part of its efforts to  update the program’s economic analysis.
Additionally, the officials stated that the $601 million does not include JR&IO’s  actual and estimated costs of $153 million through fiscal year 2009 for  requirements definition activities, business process reengineering planning,  enterprise architecture development, and other activities pertaining to management  and analysis of the human resources domain. According to JR&IO officials, this  $153 million consists of $72.5 million obligated during fiscal years 1998 through  2003 and $80.4 million required for fiscal years 2004 through 2009.
section_title: Benefits
section_title: DOD estimated that after DIMHRS (Personnel/Pay) is fully operational, including implementation of associated business process improvements and termination of legacy systems, the system will address DOD’s long-standing military personnel and payroll processing problems and result in productivity improvements and reduced IT operating and maintenance costs.4
subsections: section_title: According to DOD, mission performance improvements, not cost savings, are the rationale for DIMHRS (Personnel/Pay).
paragraphs: Objective 1: Requirements Management DOD’s management of the DIMHRS (Personnel/Pay) requirements definition  has recently improved, but key aspects of requirements definition remain a  challenge. Requirements are the foundation for designing, developing, and testing  a system. Our experience indicates that incorrect or incomplete requirements are a  common cause of systems not meeting their cost, schedule, or performance goals.  Disciplined processes and controls for defining and managing requirements are  defined in published models and guides, such as the Capability Maturity Models  developed by Carnegie Mellon University’s Software Engineering Institute (SEI),  and standards developed by the Institute of Electrical and Electronics Engineers  (IEEE).
In managing DIMHRS (Personnel/Pay) requirements, DOD has begun taking steps  to ensure that the system requirements and the system design are consistent with  each other. However, DOD  has not ensured that the detailed requirements are complete and   has not obtained user acceptance of the detailed requirements.
These challenges increase the risk that the delivered system’s capabilities will not  fully meet DOD’s needs.
See, for example, Karl E. Wiegers, Software Requirements (1999), p.15, and GAO, DOD Business  Systems Modernization: Billions Continue to Be Invested with Inadequate Management Oversight and  Accountability, GAO-04-615 (Washington, D.C.: May 27, 2004).
Objective 1: Requirements Management Traceability An accepted way of ensuring the complete and accurate incorporation of  requirements is to trace between levels of requirements and design documentation:  Document (ORD)
In system development, traceability is the degree to which a relationship is  established between two or more products of the system development process.
Traceability allows the user to follow the life of the requirement both forward and  backward through system documentation from origin through implementation.  Traceability is critical to understanding the parentage, interconnections, and  dependencies among the individual requirements. This information in turn is critical  to understanding the impact when a requirement is changed or deleted.
Objective 1: Requirements Management Traceability The DIMHRS (Personnel/Pay) ORD defines the high-level capabilities for satisfying  DOD’s mission needs.
The ORD lists the functional processes, information needs, and performance  parameters that the system is to support; an example of a functional process is  “promote enlisted member personnel.”
The detailed requirements include “use cases,” which are detailed descriptions of  the activities that the system and the end users must perform and data needed to  accomplish these activities.
For example, the functional process “promote enlisted personnel” includes  multiple use cases, such as “record enlisted member’s eligibility for promotion.”  Each use case includes (1) business rules describing the processing steps for  accomplishing the use case, such as steps for determining which members  meet the time in grade/time in service requirement for promotion; (2) references  to the applicable statutes, policies, guidance, or regulations that govern the use  case; and (3) a list of the information needed to perform the use case, such as  each person’s rank, occupation code, and promotion recommendation.
In addition, the use cases incorporate process improvements that are to introduce  efficiencies and to standardize personnel and pay processing DOD-wide.
Objective 1: Requirements Management Traceability However, when DOD accepted the first two parts of the system design, it had not  traced between the detailed requirements and the design. Rather, DOD required  the contractor to base the system design only on the high-level requirements  defined in the ORD, and DOD provided detailed requirements for information only  purposes.
According to JPMO officials, the contract was written in this way to provide the  contractor with maximum flexibility to design the system according to the  capabilities of the COTS product and thereby reduce system development and  maintenance costs. Nonetheless, JR&IO officials also stated that the detailed  requirements for DIMHRS (Personnel/Pay) should be the basis of the system  design.
Objective 1: Requirements Management Traceability In addition, DOD did not, until recently, trace between ORD requirements and  detailed requirements. As a consequence, we told DOD during the course of our  review that relevant financial and accounting standards were missing from the  detailed requirements, even though they had been included in the ORD.
JFMIP is a cooperative undertaking of the Treasury Department, Office of Management and Budget, Office of  Personnel Management, GAO, and others to improve financial management practices in government. The PMO,  managed by the Executive Director of the JFMIP, using funds provided by the Chief Financial Officers Council  agencies, is responsible for the testing and certification of COTS core financial systems for use by federal agencies  and coordinating the development and publication of functional requirements for financial management systems.
JFMIP, JFMIP Human Resources & Payroll Systems Requirements, JFMIP SR-99-5 (April 1999).
Objective 1: Requirements Management Traceability  The JFMIP PMO standard is important because it contains requirements  associated with producing accurate and complete payroll data for the financial  statements, which is relevant to DOD’s efforts to obtain “clean,” or “unqualified,”  audit opinions on its financial statements.8  An example of the JFMIP PMO requirements is functionality to process prior  period, current, and future period pay actions, on the basis of effective dates.
A clean, or unqualified, opinion is given when an auditor deems the financial statements to be accurate and  complete, with no qualifying statements.
Objective 1: Requirements Management Traceability  After we told DOD of the missing requirements, JR&IO officials undertook an  analysis of the 196 JFMIP PMO human resources and payroll requirements9 and have stated that this analysis has allowed them to ensure that DIMHRS  (Personnel/Pay) will meet 170 of the 196 requirements and that the remaining  requirements are not applicable to military human resource and payroll systems.  They also stated that all applicable requirements are now documented in the  DIMHRS (Personnel/Pay) detailed requirements database.
Objective 1: Requirements Management Traceability  According to JPMO officials, tracing the detailed financial requirements to the  design was not done sooner because the COTS software product being used is  certified as JFMIP PMO compliant. However, JFMIP PMO certification extends  only to the core financial module of this software (i.e., general ledger, funds  control, accounts receivable, accounts payable, cost management, and  reporting); it does not include the two modules used for DIMHRS  (Personnel/Pay)—human resources and payroll.
Objective 1: Requirements Management Traceability  In addition, as we have reported, even if JFMIP PMO had certified the human  resources and payroll modules of the COTS software product, certification by  itself does not ensure that systems based on this software will be compliant with  the goals of the Federal Financial Management Improvement Act, as JFMIP has  made clear, and does not ensure that systems based on this software will  provide reliable, useful, and timely data for day-to-day management.10 Other  important factors affecting compliance with federal financial management  system requirements and the effectiveness of an implemented COTS system  include how the software package has been configured to work in the agency’s  environment, whether any customization is made to the software, the success of  converting data from legacy systems to new systems, and the quality of  transaction data in the feeder systems.
GAO, Financial Management: Improved Financial Systems Are Key to FFMIA Compliance, GAO-05-20  (Washington, D.C.: Oct. 1, 2004), and Business Modernization: NASA’s Integrated Financial Management  Program Does Not Fully Address Agency’s External Reporting Issues, GAO-04-151 (Washington, D.C.:  Nov. 21, 2003).
Objective 1: Requirements Management Traceability To their credit, JR&IO and JPMO officials have begun tracing between the detailed  requirements and the design, including the financial standards. As of late  November 2004, they told us that this tracing had identified about 630  discrepancies that may require modification to the detailed requirements or the  design. They stated that they plan to complete this tracing by the end of February  2005.
Until DOD completes tracing both backward (from the design back to the detailed  requirements and the ORD) and forward (from the ORD forward to the detailed  requirements and the design), the risk is increased that the requirements and  design are not complete and correct.
Objective 1: Requirements Management Content of Requirements Detailed requirements are missing important content and are difficult to understand.  According to SEI, requirements should be complete, correct, clear, and  understandable; IEEE standards state that requirements should be communicated  in a structured manner to ensure that the customers (i.e., end users) and the  system’s developers reach a common understanding of them.
For DIMHRS (Personnel/Pay), certain requirements are missing from the detailed  requirements. Specifically, the interface requirements remain incomplete, and  questions exist as to the completeness of the data requirements. Finally, some of  the use cases that provide the detailed requirements are unclear and ambiguous,  making them difficult to understand. Each of these three areas is discussed in  greater detail below.
If requirements are not complete and clear, their implementation in the system  design may not meet users’ needs, and it will be unnecessarily difficult for DOD to  test the system effectively and determine whether system requirements have been  met.
Objective 1: Requirements Management First, the requirements for the interfaces between DIMHRS (Personnel/Pay) and  existing systems are not yet complete. According to SEI, requirements for internal  and external interfaces should be sufficiently defined to permit these interfaces to  be designed and interfacing systems to be modified.
For example, DIMHRS (Personnel/Pay) will be required to interface with DOD’s  accounting systems and other systems, such as DOD’s travel system, either by  providing DIMHRS (Personnel/Pay) data for these systems or by receiving  accounting data from them. DIMHRS (Personnel/Pay) interfaces must also be  designed to ensure compliance with applicable JFMIP PMO financial system  requirements and applicable federal accounting standards. These interface  requirements must be completed before the DIMHRS (Personnel/Pay) system can  be fully deployed.
To complete the interface requirements, officials representing JPMO and the user  organizations’ DIMHRS offices told us that they must identify which of the legacy  systems will be partially replaced, and thus will require modification in order to  interface with the new system. JPMO officials stated that although DOD accepted  the system design in November 2004, a significant amount of work remained to  fully address DIMHRS (Personnel/Pay) interface issues by the user organizations.
Objective 1: Requirements Management Content of Requirements Second, the data requirements initially provided to the contractor for the system  design had not been aligned with the users’ information needs that were included in  the detailed requirements. According to SEI, the data required to meet users’  information needs must be defined so that the system can be properly designed  and developed. However, JR&IO officials told us that they had not fully defined  information needs when users were asked to identify the data requirements, along  with the legacy systems that are the best sources of the required data. The  contractor needed these data requirements to complete the system design on  schedule.
DOD recently began comparing the data requirements provided to the  contractor with the users’ information needs developed by JR&IO. JR&IO  officials stated that they expect to complete this work in February 2005. Until  this task is completed, DOD will not know whether revisions will be needed to  the system design to ensure that users’ information needs are met and that the  correct data are later migrated to the new system.
JPMO officials also stated that when DOD accepted the system design in  November 2004, a significant amount of work remained for the user  organizations to fully address DIMHRS (Personnel/Pay) data issues.
Objective 1: Requirements Management Content of Requirements Third, some of the detailed requirements are unclear and ambiguous, making them  difficult to understand. According to SEI, requirements should be clear and  understandable. When requirements are ambiguous, their meaning is open to  varying interpretations, which increases the risk that the implementation of the  requirements will not meet users’ needs.
We reviewed a random sample of the documentation for 40 of 424 use cases  (detailed requirements): 20 of 284 pay use cases and 20 of 140 personnel use  cases. Our review showed that this documentation did not consistently provide  a clear explanation of the relationships among the parts of each requirement  (business rules, information requirements, and references) or adequately  identify the sources of data required for computations.
Based on our sample, an estimated 22 percent of use cases cite “P&R  guidance” (that is, guidance from the Office of the Under Secretary for P&R) as  a reference to support the need for a business rule, either alone or along with  references to DOD policies.11 According to JR&IO officials, this note indicates  that the business rule includes steps not currently required by DOD’s policies,  which have been added either to take advantage of “out-of-the-box” COTS  capabilities or to implement a best practice. However, when P&R guidance is  cited, the use cases do not explain whether an out-of-the-box capability or best  practice is intended.
This estimate is a weighted average of the sample results for the two categories of use cases shown in  the table on slide 34. A weighted average is used because the population of pay use cases was sampled at  a rate different from the population of personnel use cases.
Objective 1: Requirements Management Content of Requirements  In addition, when both P&R guidance and existing policies are cited, the use  case does not explain which rules are based on P&R guidance and which are  based on existing policies. These ambiguities make it difficult for stakeholders to  understand the business rule and its rationale. (This point is further discussed in  the following section on end users’ acceptance of requirements.) According to  JR&IO officials, such ambiguity was resolved via communication between  JR&IO and JPMO officials followed by JPMO officials’ communicating with the  contractor.
Estimates of the extent of use case problems and associated confidence  intervals are summarized on the next slide.
section_title: Extent of Problems with Use Cases
subsections: section_title: Problem description Information requirements are provided, but the business rules do not provide enough detail to determine what information requirement is to be used for each business rule. Business rule contains a computation but does not include a source for the data needed for making the computation. Reference to “P&R guidance” does not clearly explain the rationale behind the business rule. Use cases that had one or more of the above problems
paragraphs: Objective 1: Requirements Management Content of Requirements JR&IO officials agreed that the clarity of the use cases could be improved but  stated that the use cases provide a greater level of detail than DOD normally  provides for a COTS-based system. JR&IO officials added that they developed the  use cases to support the design and development of the system rather than to  communicate the detailed requirements to the user organizations. In this regard,  officials representing the DIMHRS (Personnel/Pay) development and integration  contractor stated that the use cases are providing useful information for designing  the DIMHRS (Personnel/Pay) system.
Objective 1: Requirements Management DOD has not obtained user acceptance of detailed requirements. According to SEI,  users’ needs and expectations must be used in defining requirements.  Furthermore, according to our guidance, when business process changes are  planned, users’ needs and expectations must be addressed, or users may not  accept change, which can jeopardize the effort.12 One way to ensure and  demonstrate user acceptance of requirements is to obtain sign-off on the  requirements by authorized end-user representatives.
To its credit, JR&IO obtained the user organizations’ formal acceptance of the  ORD. However, the process used to define the detailed requirements (specifically,  the use cases) has not resulted in user acceptance.
End-user representatives stated that their involvement in the definition of the  use cases was limited.
End users’ comments on the use cases were not fully resolved.
End users are not being asked to approve the detailed requirements.
Each of these three issues is discussed in greater detail below.
GAO, Business Process Reengineering Assessment Guide Version 3, GAO/AIMD-10.1.15 (Washington,  D.C.: May 1997).
Objective 1: Requirements Management End User Acceptance First, JR&IO developed the use cases with the help of contractors and  representative end users (personnel and payroll specialists) from the end-users’  stakeholder organizations. However, according to these representatives, their role  in defining the use cases was limited. They stated that they principally performed  research, identified references, and explained how their legacy environments  currently process personnel and pay transactions and that they had limited  influence in deciding the content of the use cases.
In response, JR&IO officials stated that many of the user representatives were  lower-level personnel who were not empowered to represent their components in  making decisions about requirements.
Objective 1: Requirements Management End User Acceptance Second, to obtain users’ comments on the use cases, JR&IO provided the end- users’ organizations with the use cases and other documentation for comment, but  this process did not resolve all of the comments.
An initial set of use cases was reviewed by hundreds of individual end users  (personnel and payroll specialists), resulting in thousands of comments.
Around October 2003, JR&IO provided a second set of use cases to the end  users (as well as to the development and integration contractor), including  modifications to reflect changes suggested by users based on their first set of  comments. The end users provided about 7,000 comments on the second set of  use cases.
Objective 1: Requirements Management End User Acceptance  In March 2004, JR&IO established a baseline version of the use cases and  provided this version to the development and integration contractor in order to  meet the contractor’s schedule. At that time, JR&IO had concurred with about  400 of the 7,000 comments and modified the use cases in response, but it had  not completed its analysis of all 7,000 comments.
JR&IO then established a change control process for making further  modifications to the baseline use cases. According to JR&IO officials, as of the  end of October 2004, 703 change requests had been submitted: 163 had been  approved, 48 had been disapproved, and the remaining 492 were still under  review.
Objective 1: Requirements Management End User Acceptance  According to end-user representatives from each of the services, the use cases  were difficult to understand because they were shared in a piecemeal fashion  and did not include sufficient detail. Furthermore, they said that JR&IO  responses to comments were generally brief and often did not provide sufficient  explanation—for example, “The Business Rule captures this requirement.  Action: No change required…” and “The comment is out of scope. Action: No  action required.” As a result, the end-user representatives stated that they often  did not understand the reasoning behind the decisions.
JR&IO officials stated that users might have had difficulty with understanding  the use cases because they were defined in terms of what processes the  system would perform as opposed to how the processes would be performed.  This approach is consistent with best practices (as discussed later in this  briefing) and with JR&IO’s stated intention to discourage the definition of  processes in terms of existing systems and processes, as well as to allow the  development of reengineered joint processes using the native capabilities of the  COTS software to the maximum extent. However, best practices also require  that explanations of business rules and their rationale be complete and  understandable by end users.
Objective 1: Requirements Management End User Acceptance  JR&IO officials further stated that many of the end users’ comments were not  substantive (e.g., a minor change needed in the citation of a regulation); many  were duplicative, and some addressed issues outside of personnel/pay  functionality, such as training and manpower. In addition, most were not  prioritized.
Objective 1: Requirements Management End User Acceptance Third, JR&IO does not intend to gain formal agreement on the detailed  requirements from the end-users’ organizations, although it did obtain such  agreement on the ORD. JR&IO officials stated that gaining formal agreement from  some of the users’ organizations would delay the program and be impractical  because of some user organizations allegiance to their legacy systems and  processes.
GAO, Defense IRM: Poor Implementation of Management Controls Has Put Migration Strategy at Risk,  GAO/AIMD-98-5 (Washington, D.C.: Oct. 20, 1997).
GAO, Information Technology: DOD's Acquisition Policies and Guidance Need to Incorporate Additional  Best Practices and Controls, GAO-04-722 (Washington, D.C.: July 30, 2004).
Nevertheless, not attempting to obtain agreement on DIMHRS  (Personnel/Pay) requirements increases the risk that users will not accept and  use the developed and deployed system, and that later system rework will be  required to make it function as intended and achieve stated military human  capital management outcomes.
Officials representing the DIMHRS offices are not in full agreement with JR&IO  officials on the state of the requirements, as the following slides show.
Objective 1: Requirements Management End User Acceptance Officials representing each of the DIMHRS management offices (Army, Air Force,  Navy, Marine Corps, and DFAS) stated that their organizations are not currently  willing to sign off on the DIMHRS (Personnel/Pay) detailed requirements as being  sufficient to meet their organizations’ military personnel and pay needs. These  officials stated that they do not yet know what the gaps are between the  functionality provided by their current systems and the functionality to be provided  by DIMHRS (Personnel/Pay).
Officials representing the Army’s DIMHRS office stated that they do not yet  know whether the requirements are adequate to enable the Army to replace a  number of its existing systems with DIMHRS (Personnel/Pay).
Officials representing the Air Force’s DIMHRS office stated that the  requirements are defined at a very high level and are subject to interpretation,  and as a result, they are unable to determine whether DIMHRS (Personnel/Pay)  will meet all the Air Force’s requirements.
Objective 1: Requirements Management End User Acceptance  Officials representing the Navy’s DIMHRS office stated that their concerns  about the adequacy of the detailed requirements relate principally to the issue of  not knowing which of the functions provided by the Navy’s legacy systems will  not be provided by DIMHRS (Personnel/Pay).
Officials representing the Marine Corps’ DIMHRS office stated that they do not  believe that the requirements adequately address service specificity or the  automation of manual processes. These officials stated that the Marine Corps  has not been able to determine what functionality the system contains versus  the functionality contained in its legacy systems. They stated that this  information is needed to enable them to modify legacy systems to ensure that  needed functions continue to be provided to service members until these  functions are incorporated into DIMHRS (Personnel/Pay).
Officials representing DFAS’s DIMHRS office stated that they do not believe the  requirements adequately address a number of pay, accounting, and personnel  issues.
According to JR&IO officials:  DIMHRS (Personnel/Pay) will provide all the functionality provided by the  services’ and DFAS’ legacy personnel and pay systems. JR&IO stated that the  defined requirements provide enough information to determine what the system  will do but acknowledged that understanding exactly how the system would  perform its functions was not possible until the system was fully designed.
Owners of end users’ legacy systems are generally not supportive of DIMHRS  (Personnel/Pay) because they want to preserve their autonomy in the  development and control of their own systems.
Support for the system by the services’ executives is mixed. For example,  JR&IO said that (1) Army executives are committed to implementing and using  the DIMHRS (Personnel/Pay) system because they believe it will address many  problems that the Army currently faces, (2) Air Force officials are generally  supportive of the system but say they do not yet know whether the system will  meet all their needs, and (3) Navy and Marine Corps executives are not as  supportive because they are not fully convinced that DIMHRS (Personnel/Pay)  will be an improvement over their existing systems.
Objective 1: Requirements Management End User Acceptance  A number of actions have been taken to reduce the risk that users will not  accept the system, including conducting numerous focus groups, workshops,  demonstrations, and presentations explaining how the DIMHRS  (Personnel/Pay) system could address DOD’s existing personnel/pay problems.
Although JPMO officials told us that a consensus of the users’ organizations  agreed with the decision to accept the contractor’s design of DIMHRS  (Personnel/Pay) in November 2004, they submitted 391comments and issues on  the design. JPMO officials stated that they expect to resolve these comments and  issues by the end of February 2005.
Objective 2: Program Management  DOD does not have a well-integrated management structure for DIMHRS  (Personnel/Pay) and is not following all relevant supporting acquisition  management processes. In particular,  program responsibility, accountability, and authority are diffused;   the system has not been defined and designed according to a DOD-wide  integrated enterprise architecture;15  program stakeholders’ activities have not been managed according to a master  plan/schedule that integrates all stakeholder activities; and  the program is following some, but not all, best practices associated with  acquiring business systems based on commercially available software.
Without a well-integrated approach and effective processes for managing a  program that is intended to be an integrated solution, DOD has increased the risk  that the program will not meet cost, schedule, capability, and outcome goals.
JPMO officials stated that the system has not been defined and designed according to a DOD-wide  integrated enterprise architecture because the enterprise architecture is not complete.
section_title: Objective 2: Program Management Structure and Processes Governance
paragraphs: Program responsibility, accountability, and authority are diffused. Research shows  that leading organizations ensure that programs are structured to ensure that  assigned authorities, responsibilities, and accountabilities are clear and aligned  under the continuous leadership and direction of a single entity. For DIMHRS  (Personnel/Pay), these areas are spread among three key stakeholder groups  whose respective chains of command do not meet at any point below the Secretary  and Deputy Secretary of Defense level.
Responsibility for requirements definition rests with JR&IO, which is  accountable through one chain of command.
Responsibility for system acquisition rests with JPMO, which is accountable  through another chain of command.
Responsibility for preparing for transition to DIMHRS (Personnel/Pay) rests with  the end users’ organizations—11 major DOD components reporting through five  different chains of command.
The organization chart on the next slide shows the chain of command and the  coordination relationships among the primary DIMHRS (Personnel/Pay)  stakeholder groups.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: Assist.
Reserves)
Reserves)
Reserves)
Reserves) (R,D,& A)
section_title: Objective 2: Program Management Structure and Processes
paragraphs: As the chart also shows, the services and DFAS have DIMHRS (Personnel/Pay)  management offices to assist JR&IO and JPMO and to represent their respective  end-user communities (the pay and personnel specialists). In addition, various  coordination and advisory bodies have been established.
The three primary stakeholder groups (JR&IO, JPMO, and the end users) are  accountable to three different groups of executives.
JR&IO is ultimately accountable to the Under Secretary for P&R, who is the  department’s Principal Staff Assistant (PSA)16 for personnel and compensation  and is responsible for oversight of the DIMHRS (Personnel/Pay) program from a  functional perspective.
The PSA is the executive-level manager responsible for the management of defined functions within  DOD.  According to JR&IO officials, the separation of the functional and acquisition lines of authority is a normal  DOD practice.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: The end users are ultimately accountable to the Offices of the Secretaries of the  Army, Air Force, and Navy (in coordination with the Commandant of the Marine  Corps) and the DOD Comptroller, who have ultimate responsibility for  implementing and using DIMHRS (Personnel/Pay).
See, for example, GAO/AIMD-98-5.
section_title: Objective 2: Program Management Structure and Processes Governance
paragraphs: Although no stakeholder organization has continuous programwide oversight  purview and visibility, the DIMHRS (Personnel/Pay) Executive Steering Committee  is made up of representatives of each of the entities that has ultimate responsibility  for the program.
According to DOD, the committee monitors the program, resolves issues that are  brought before it, and advises the Under Secretary (P&R).19 It meets quarterly or  when assembled by the chair—the Deputy Under Secretary of Defense for  Program Integration—who reports to the Under Secretary (P&R).
DIMHRS (Pers/Pay) Report to Congress (June 2002).
section_title: Objective 2: Program Management Structure and Processes
paragraphs: According to JR&IO officials, the diffusion of program accountability, responsibility,  and authority will be reduced in fiscal year 2005, when funding for both JR&IO and  JPMO will be consolidated and centrally managed by JR&IO.20 However, the end- user organizations will continue to separately control their respective funds.
For example, officials at the Army’s DIMHRS (Personnel/Pay) management  office estimated that the Army’s DIMHRS (Personnel/Pay) funding needs will  range from $27 million to $43 million a year, but they said that the Army is  unlikely to fund the program at that level because of other priorities.
Without a DOD-wide integrated governance structure that vests an executive-level  organization or entity representing the interests of all program stakeholders with  responsibility, accountability, and authority for a joint or integrated program like  DIMHRS (Personnel/Pay), DOD runs the risk that the program will not produce an  integrated set of outcomes.
According to JR&IO officials, DOD requested consolidated JR&IO and JPMO funding for DIMHRS  (Personnel/Pay) for fiscal year 2005.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: GAO, Information Technology: A Framework for Assessing and Improving Enterprise Architecture  Management (Version 1.1), Executive Guide, GAO-03-584G (Washington, D.C.: April 2003).
Bob Stump National Defense Authorization Act for Fiscal Year 2003, Pub. L. No. 107-314, section 1004,  116 Stat. 2458, 2629–2631 (Dec. 2, 2002).
section_title: Objective 2: Program Management Structure and Processes
paragraphs: Acquiring and implementing DIMHRS (Personnel/Pay) without an enterprise  architecture increases the risk that DOD will make a substantial investment in  system solutions that will not be consistent with its eventual blueprint for business  operational and technological change.
Recognizing this, the Deputy Secretary of Defense issued a memorandum in  March 2004 requiring the development and implementation of architectures for  each of DOD’s six business domains, including the human resources domain.  These business domains, according to the department’s modernization program,  are delegated the “authority, responsibility, and accountability … for their respective  business areas” for implementing business transformation,23 including the following:  “Leading the business transformation within the Domain.”   “Managing its respective  portfolio to ensure implementation  of and compliance with the Business Enterprise Architecture (BEA) and  transition plan.”   “Assisting in the extension of the  BEA” for the domain.
DOD Business Management Modernization Program, Governance Approach.  http://www.dod.mil/comptroller/bmmp/pages/govern_dod.html.
section_title: Objective 2: Program Management Structure and Processes Enterprise Architecture
paragraphs: The Under Secretary (P&R), who is the domain owner for the human resources  domain, assigned JR&IO the responsibility for extending the BEA for the human  resources domain. According to JR&IO officials, the development of the human  resources portion of the BEA is being done concurrently with the acquisition and  deployment of DIMHRS (Personnel/Pay).
section_title: Objective 2: Program Management Structure and Processes
paragraphs: Recognizing the importance of managing the concurrency of such activities and  ensuring that DOD’s ongoing investments are pursued within the context of its  evolving BEA, the National Defense Authorization Act for Fiscal Year 2003 also  required that system improvements with proposed obligations of funds greater than  $1 million be reviewed to determine if they are consistent with the BEA.
To satisfy this requirement, JPMO officials presented the DOD Office of the  Comptroller, which is developing the BEA, with information on DIMHRS  (Personnel/Pay) compliance with version 1.0 of the BEA in April 2003. However,  according to our review of the information used by JPMO in April 2003 to obtain an  architectural compliance determination, this information did not include a  documented, verifiable analysis demonstrating such compliance. In the absence of  such analysis, the JPMO program manager instead made commitments that  DIMHRS (Personnel/Pay) would be consistent with the architecture. On the basis  of this commitment, the DOD Comptroller certified in April 2003 that DIMHRS  (Personnel/Pay) is consistent with the BEA. Later, JPMO included in the DIMHRS  (Personnel/Pay) contract a requirement that the systems specification be  compatible with the emerging BEA.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: According to JR&IO officials, the April 2003 architectural certification is preliminary,  and further certification is needed. They stated that DIMHRS (Personnel/Pay) will  undergo another certification before the system deployment decision.
By this time, however, lengthy and costly DIMHRS (Personnel/Pay) design and  development work will be completed. The real value in having and using an  architecture is knowing, at the time that extensive system definition, design, and  development are occurring, what the larger blueprint for the enterprise is, so that  definition, design, and development can be guided and constrained by this frame of  reference. Aligning to the architecture after the system is designed could also  require expensive system rework to address any inconsistencies with the  architecture.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: The absence of verifiable analysis of compliance was in part due to the state of  completeness of BEA version 1.0. As we reported in September 2003, this version  was missing key content, including sufficient depth and detail to effectively guide  and constrain system investments.24 Since then, DOD has issued other versions of  the BEA. However, we reported in May 2004 that version 2.0 of the BEA still did not  include many of the key elements of a well-defined architecture.25 For example, the  “to be” environment did not provide sufficient descriptive content related to future  business operations and supporting technology to permit effective acquisition and  implementation of system solutions and associated operational change. DOD since  issued versions 2.2 and 2.2.1 in July and August 2004, respectively.
GAO, DOD Business Systems Modernization: Important Progress Made to Develop Business Enterprise  Architecture, but Much Work Remains, GAO-03-1018 (Washington, D.C.: Sept. 19, 2003).
GAO, DOD Business Systems Modernization: Limited Progress in Development of Business Enterprise  Architecture and Oversight of Information Technology Investments, GAO-04-731R (Washington, D.C., May  17, 2004).
section_title: Objective 2: Program Management Structure and Processes
paragraphs: DIMHRS (Personnel/Pay) program stakeholder activities are not being managed  according to an integrated master plan/schedule. IEEE standards state that a  master plan/schedule should be prepared and updated throughout the system’s life  cycle to establish key events, activities, and tasks across the program, including  dependencies and relationships among them. A properly designed master  plan/schedule should allow for the proper scheduling and sequencing of activities  and tasks, allocation of resources, preparation of budgets, assignment of  personnel, and criteria for measuring progress.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: making revisions to their regulations to ensure consistency with the  reengineered business rules designed into DIMHRS (Personnel/Pay) that differ  from existing DOD or service rules and policies (e.g., those noted earlier in this  briefing as being in accordance with “P&R guidance” in the use cases).
With a plan/schedule that focuses on the contractor’s and JPMO’s activities and  does not extend to all DOD program stakeholders’ activities, the risk increases that  key and dependent events, activities, and tasks will not be performed as needed,  which in turn increases the risk of schedule slippage and program goal shortfalls.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: GAO-04-722.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: ensuring that plans explicitly provide for preparing users for the impact that the  business processes embedded in the commercial components will have on their  respective roles and responsibilities,  proactively managing the introduction and adoption of changes to how users will  be expected to use the system to execute their jobs, and  ensuring that project plans explicitly provide for the necessary time and  resources for integrating commercial components with legacy systems.
To its credit, DOD is following the first three of these practices for DIMHRS  (Personnel/Pay), but its is not following the last three. For example, program  officials told us that they expected the contractor to base the system design on the  high-level requirements defined in the ORD as a way to maximize the contractor’s  ability to leverage the COTS product. Furthermore, the contract includes award  fees that give the contractor incentives to, among other things, minimize  customization of the COTS software.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: However, DOD does not have an integrated program plan/schedule that provides  for end-user organization activities that are associated with preparing users for the  operational and role-based changes that the system will introduce, such as the  need to revise the duties that are now performed by pay specialists and personnel  specialists. Furthermore, DOD’s program plans do not recognize the end-user  organizations’ time and resource needs associated with integrating DIMHRS with  their respective legacy systems, and JPMO is not actively managing these end- user operational changes. Although JR&IO officials told us that some planning has  occurred to position end users for DIMHRS (Personnel/Pay) changes, officials  representing the DIMHRS offices in the services and DFAS stated that these plans  do not adequately address the above areas.
section_title: Objective 2: Program Management Structure and Processes
paragraphs: By not following all relevant best practices associated with acquiring COTS-based  systems, DOD is increasing the risk that DIMHRS (Personnel/Pay) will not be  successfully implemented and effectively used.
section_title: The importance of DIMHRS (Personnel/Pay) to DOD’s ability to manage military personnel and pay services demands that the department employ effective processes and structures in defining, designing, developing, and deploying the system to maximize its chances of success. For DIMHRS (Personnel/Pay), however, DOD did not initially perform important requirements development steps, and the detailed system requirements are missing important content. DOD has begun to remedy these omissions by taking actions such as tracing among requirements documents and system design documents to ensure alignment, but user organizations’ acceptance of requirements has not occurred. Moreover, although DIMHRS (Personnel/Pay) is to be an integrated system, it is not being governed by integrated tools and approaches, such as an integrated program management structure, integrated DOD business enterprise architecture, and an integrated master plan/schedule.
section_title: It is critical that DOD carefully consider the risks caused by each of these areas of concern and that it appropriately strengthen its management processes, structures, and plans to effectively minimize these risks. To do less undermines the program’s chances of timely and successful completion.
section_title: To assist DOD in strengthening its program management processes, structures, and plans and thereby increase its chances of successfully delivering DIMHRS (Personnel/Pay), we recommend that the Secretary of Defense direct the Assistant Secretary (NII), the Under Secretary (P&R), and the Under Secretary (Comptroller), in collaboration with the leadership of the military services and DFAS, to jointly ensure an integrated, coordinated, and risk-based approach to all DIMHRS (Personnel/Pay) definition, design, development, and deployment activities. At a minimum, this should include
paragraphs: ensuring that all relevant acquisition management best practices associated  with COTS-based systems are appropriately followed; and  adopting a more event-driven, risk-based approach to managing DIMHRS that  adequately considers factors other than the contract schedule.
Our objectives were to determine 1. whether the Department of Defense (DOD) has effective management processes in place for managing the definition of the requirements for the  Defense Integrated Military Human Resources System (DIMHRS  (Personnel/Pay)) and 2. whether DOD has established an integrated program management  structure for DIMHRS (Personnel/Pay) and is following effective processes  for acquiring a system based on commercial software components.
To determine industry and government best practices and regulations for effective  requirements definition and management, we evaluated criteria from the Capability  Maturity Models (CMM) developed by Carnegie Mellon University’s Software  Engineering Institute and standards developed by the Institute of Electrical and  Electronics Engineers (IEEE), as well as the DOD 5000 series and other applicable  DOD policies and regulations, federal accounting standards, and prior GAO reports  and best practices guidance.
Objectives, Scope, and Methodology   discussed the use cases with selected personnel and pay specialists and  reviewed end users’ written comments to JR&IO on the use cases;   estimated the extent of several problems by evaluating the clarity and  understandability of use cases in a probability sample27 of 20 of 284 pay use  cases and 20 of 140 personnel use cases; and   compared requirements management activities with relevant industry and  government guidance and requirements, including CMM and IEEE, and the  DOD 5000 series and Joint Chiefs of Staff regulations.
A different probability sample of use cases could produce different estimates. In this briefing, we present  estimates along with the 95 percent confidence intervals for these estimates. This means that there is a 95  percent probability that the actual value for the entire population is within the range defined by the  confidence interval. In other words, if 100 different samples were taken, in 95 of those 100 samples, the  actual value for the entire population would be within the range defined by the confidence interval, and in 5  of those 100 samples, the value would be either higher or lower than the range defined by the confidence  interval.
To evaluate program management structures and processes, we   interviewed officials from JR&IO, JPMO, and the DIMHRS offices for each of the   analyzed DOD and DIMHRS (Personnel/Pay) program management and  process management documentation and activities, including charters, process  descriptions, budgets, and program plans, etc.; and  reviewed relevant analysis supporting program decisions, such as economic  justification and architectural alignment.
We determined that the data used in this report are generally reliable for the  purposes for which we used them. For DOD-provided data, we have made  appropriate attribution to indicate the data’s source.
We performed our work at DOD headquarters; JR&IO, in the Washington, D.C.,  area; JPMO in New Orleans, Louisiana; the Army’s, Navy’s, Air Force’s, and  Marine Corps’ DIMHRS offices; and DFAS’s offices in the Washington, D.C., area.
This work was performed from January through November 2004 in accordance with  generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
paragraphs: The following are GAO’s comments on the Department of Defense’s letter  dated January 25, 2005.
subsections: section_title: GAO’s Comments
paragraphs: 1.  The Department of Defense’s (DOD) characterization of our objectives  is not correct. As stated in our report, our objectives were to  determine whether DOD had effective processes in place for managing  the definition of requirements for the Defense Integrated Military  Human Resources System (DIMHRS) (Personnel/Pay) and whether it  established an integrated program management structure and followed  effective processes for acquiring a system based on commercial  software components. Accordingly, we assessed the processes used to  manage DIMHRS (Personnel/Pay), and the content of the  requirements, against relevant best practices, many of which are  embodied in DOD and federal policies and guidance, and against  federal guidance, federal accounting standards, and prior GAO reports.   2.  We do not believe that our finding that DOD is appropriately limiting  modification of commercial, off-the-shelf (COTS) products (a best  practice) is incongruous with our recommendation that requirements  be acceptable to user organizations (another best practice).  Furthermore, our report does not recommend that DOD act on all  comments regardless of impact. Our recommendations concerning  system requirements are intended to provide DOD with the principles  and rules that it should apply in executing a requirements-acceptance  process that permits all stakeholder interests and positions to be  heard, considered, and resolved in the context of what makes  economic sense. Furthermore, our report makes complementary  recommendations that discourage changes to COTS products unless  fully justified on the basis of life-cycle costs, benefits, and risks.  Finally, while we do not dispute whether DOD has followed a process  to screen out comments that would have necessitated COTS  modification, DIMHRS (Personnel/Pay) users said that this process did  not allow for effective resolution of the comments, which is the basis  for our recommendation aimed at gaining user acceptance of  requirements.  3.  We have not concluded that DOD had not done enough to ensure that  all stakeholders have had full input to the requirements. Our  conclusion was that DOD had not obtained user acceptance of the  detailed requirements, and that this choice entails risks.  4.  We disagree. Our report neither states nor suggests that DOD act on all  comments that it receives on requirements from all sources. Also, see  comment 2.  5.  See comment 10.  6.  We acknowledge DOD’s implementation of certain best practices as  noted in our report. However, at the time we concluded our work,  DOD was not following all relevant and practicable best practices, as  we discuss in our report.  7.  We do not dispute DOD’s comment about efforts on DIMHRS  (Personnel/Pay) relative to other system acquisitions because our  review’s objectives and approach did not extend to comparing the two.  See comment 1 for a description of our objectives. Furthermore, while  it is correct that DOD’s regulations only require stakeholder agreement  with the Operational Requirements Document, our evaluation was not  limited to whether DOD was meeting its own policy; we also evaluated  whether DOD’s processes were consistent with industry and  government best practices.  8.  We do not disagree that DOD has taken important steps to meet the  goals of requirements completeness and correctness, and we do not  have a basis for commenting on whether the department might have  completed important requirements-to-design traceability steps since  we completed our work. However, as we state in our report, these  tracing steps began in response to the inquiries we made during the  course of our review. Furthermore, DOD’s comments contain no  evidence to show that it has addressed the limitations in the  requirements’ completeness and correctness that we cite in the report,  such as those relating to the interface and data requirements, and they  do not address the understandability issues we found relative to an  estimated 77 percent of the detailed requirements. Moreover, DOD  even stated in its comments that its latest program review revealed 606  business process comments and 17 interface comments that it deemed  noncritical, although it noted that they were still being analyzed.  9.  We do not dispute DOD’s position that the Joint Requirements  Oversight Council’s validation of the Operational Requirements  Document is all that is required by DOD regulation, and we do not  have a basis for commenting on whether its documentation of  requirements for DIMHRS (Personnel/Pay) was innovative and  unprecedented. Our review objective relating to requirements, as  stated in our report, was to address whether DOD has effective  processes in place for managing the definition of the requirements. To  accomplish this objective, and as also stated in our report, we analyzed  DOD’s requirements management efforts against recognized best  practices.  10.  We do not disagree that DOD has taken numerous steps to gain user  acceptance of the system. However, as we point out in the report, user  organizations still had questions and reservations concerning the  requirements. Not adequately resolving these issues, and thereby  gaining user acceptance of requirements, increases the risk that a  system will be developed that does not meet users’ needs, that users  will not adopt the developed and deployed system, and that later  system rework will be required to rectify this situation.  11.  We do not question that DOD has reviewed the detailed requirements  since we completed our review. However, we challenge DOD’s  comment that all questions have been resolved for two reasons. First,  DOD’s comments contain no evidence to show that it has addressed  the limitations in the requirements’ completeness and correctness that  we cite in the report, such as those relating to the interface and data  requirements, or the understandability issues we found relative to an  estimated 77 percent of the detailed requirements. Second, in its  comments, DOD acknowledges that 606 questions remain regarding  requirements and design issues.  12.  We agree that the detailed requirements are not the sole vehicle for  gaining user acceptance of the system. Rather, they are one vehicle to  be used in a continuous process to ensure acceptability of a system to  end users. Industry and government best practices advocate users’  understanding and acceptance of requirements, and these practices are  not limited to high-level requirements descriptions, but rather apply to  more detailed requirements descriptions as well.  13.  We do not dispute that the roles and responsibilities of the Executive  Steering Committee are defined and documented. In fact, we cite the  committee’s responsibilities in our report.  14.  We agree that DOD has forums and processes for communicating  stakeholder interests. However, we do not agree that these have  provided for effective resolution of concerns and comments, as we  describe in our report. Also, see comment 10.  15.  We disagree. In our view, any entity, whether it is an individual, office,  or committee, can have responsibility, accountability, and authority for  managing a program. Moreover, we intentionally worded the  recommendation so as not to prescribe what entity should fulfill this  role for DIMHRS (Personnel/Pay). Rather, our intent was to ensure  that such an entity was designated and empowered.  16.  We disagree. DIMHRS (Personnel/Pay) is a DOD-wide program  involving three distinct stakeholder groups whose respective chains of  command do not meet at any point below the Secretary and Deputy  Secretary of Defense levels. As we state in our report, responsibility,  authority, and accountability for DIMHRS (Personnel/Pay) is in fact  diffused among three stakeholder groups with responsibility for  requirements with the Joint Requirements and Integration Office,  responsibility for acquisition with the Joint Program Management  Office, and responsibility for transition to DIMHRS (Personnel/Pay)  with the 11 end users’ organizations. Furthermore, under the current  structure, only one of the three stakeholder groups, the Joint  Requirements and Integration Office (JR&IO), is accountable to the  Under Secretary, and authority over DIMHRS (Personnel/Pay) is  spread across the three groups. Accordingly, the intent of our  recommendation is for DOD to create an accountability structure that  can set expectations for stakeholders and hold them accountable.  17.  See comment 2. Furthermore, we agree that the department should not  replicate “as is” processes. However, as our report points out, the  users’ comment process did not provide for effective resolution;  therefore, users stated that they were not willing to sign off on the  requirements as sufficient to meet their needs. The intent of our  recommendation is to ensure that the system’s functional acceptability  to users is reasonably ensured before the system is developed, thereby  minimizing the risk of more expensive system rework to meet users’  needs.  18.  We do not believe and nowhere in our report do we state or suggest  that stakeholders should be granted veto power. Also, see comments 2  and 3.  19.  We disagree. As stated in our report, DIMHRS (Personnel/Pay) had a  preliminary architectural certification with the Business Enterprise  Architecture (BEA) in April 2003. However, DOD could not provide us  with documented, verifiable analysis demonstrating this consistency  and forming the basis for the certification, in part because the BEA  was incomplete. Rather, we were told that this certification was based  on the DIMHRS (Personnel/Pay) program manager’s stated  commitment to be consistent at some future point, and the system is  scheduled to undergo another certification before the system  deployment decision. Moreover, we had previously reported that the  BEA, including the military personnel and pay portions of the  architecture, was not complete, and thus not in place to effectively  guide and constrain system investments. As we state in our report, the  real value in having an architecture is knowing, at the time when  system definition, design, and development are occurring, what the  larger blueprint for the enterprise is in order to guide and constrain  these activities.  20.  We disagree. See comment 21.  21.  We disagree. As we state in our report, DOD is not following three  relevant best practices. These practices are focused on effectively  planning for the full complement of activities that are needed to  prepare an organization for the institutional and individual changes  that COTS-based system solutions introduce. Such planning is  intended to ensure, among other things, that key change management  activities, including the dependencies among these activities, are  defined and agreed to by stakeholders, including ensuring that  adequate resources and realistic time frames are established to  accomplish them. In this regard, DOD agreed in its comments that it  does not have an integrated master plan/schedule for the program,  which is an essential tool for capturing the results of the proactive  change management planning that the best practices and our  recommendation advocate. Moreover, available plans did not include  all activities that end-user organizations will need to make regarding  organizational changes and business process improvements associated  with the system, such as revising the duties that are now performed by  pay specialists and personnel specialists. This concern was stated by  representatives of the DIMHRS (Personnel/Pay) offices in the services  and DFAS, who stated that current plans do not adequately address the  activities, time frames, and resources they will need to complete the  transition to DIMHRS (Personnel/Pay). Furthermore, at the time that  we completed our review, DOD had yet to identify all the legacy  systems that would interface with DIMHRS (Personnel/Pay), and so  DOD could not estimate the time and resources that will be needed to  develop and implement legacy system interfaces with DIMHRS  (Personnel/Pay). Both published research and our experience in  evaluating the acquisition and implementation of COTS-based system  solutions show that the absence of well-planned, proactive  organizational and individual change management efforts can cause  these system efforts to fail.   22.  See comment 21. Furthermore, among the ambiguities in the detailed  requirements that we cite in our report are references that do not  clearly state the associated practice or policy.  23.  See comments 8 and 22. In addition and as stated in our report, the  process used to define detailed requirements has yet to result in user  acceptance. Specifically, according to end-user representatives from  each of the services, the detailed requirements were difficult to  understand because they were shared in a piecemeal fashion and did  not include sufficient detail. Furthermore, these representatives stated  that they were not willing to sign off on the requirements.  24.  We do not dispute that DOD has provided demonstrations of and  training on the COTS product. However, as we point out in our report,  users still have questions on DIMHRS (Personnel/Pay), which will be  based on the COTS product, including how it will be used to perform  personnel and pay functions, and how it will change the roles and  responsibilities of end users. Moreover, proactive management of the  organizational and individual change associated with COTS-based  system solutions requires careful planning for the full range of  activities needed to facilitate the introduction and adoption of the  system, and as we state in the report and DOD agreed in its comments,  the department does not have the kind of integrated master plan that  would reflect such planning.   25.  We do not dispute that existing plans have been reviewed and  approved by the contractor and an independent reviewer. However,  we disagree that these plans sufficiently incorporate all the change  management activities that are needed to position DOD for adoption  and use of DIMHRS (Personnel/Pay). In the absence of an integrated  master schedule, which the department acknowledges in its comments  as yet to be developed, DOD cannot adequately ensure that the full  range of organizational and individual change management activities  will be effectively performed.  26.  We support DOD’s stated commitment to follow a more event-driven  risk-based approach, and have slightly modified our recommendation  to recognize this commitment. Nevertheless, it is important to note  that the approach that we found the department taking during the  course of our review was schedule driven, meaning that program  activities were truncated or performed concurrently in order to meet  established deadlines. For example, as we describe in our report, data  requirements (which are derived from higher-level information needs)  were provided to the contractor before information needs were fully  defined because the contractor needed these data requirements to  complete the system design on schedule. Also during our review, the  program had developed plans for accelerating system deployment in  order to meet an externally imposed deadline. After we raised concern  about the risks of accelerating the schedule, and the lack of adequate  risk-mitigation strategies, DOD changed its plans.   27.  At the time of our review, we observed that the contract was a driver  for the schedule. For example and as our report states, in March 2004,  JR&IO provided a version of the detailed requirements to the  development and integration contractor in order to meet the  contractor’s schedule, even though it had received thousands of  comments on the requirements from users that it had yet to examine  and resolve.   28.  We support DOD’s comment that it will revise the schedule if events  do not occur as anticipated because it is consistent with our  recommendation.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the person named above, the following persons made key  contributions to this report: Nabajyoti Barkakati, Harold J. Brumm,  Barbara S. Collier, Nicole L. Collier, George L. Jones, John C. Martin,  Kenneth E. Patton, B. Scott Pettis, Mark F. Ramage, Karl W. D. Seifert,  Robert W. Wagner, Joseph J. Watkins, and Daniel K. Wexler.
section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs