Department of
 

Veterans Affairs
 

Audit of
 

the Project Management
 

Accountability System
 

Implementation
 

OFFICE OF AUDITS AND EVALUATIONS

August 29, 2011 

10-03162-262 
ACRONYMS AND ABBREVIATIONS
 

CoreFLS Core Financial and Logistics System 

FLITE Financial and Logistics Integrated Technology Enterprise 

IPT Integrated Project Team 

IT Information Technology 

ITOC Information Technology and Oversight Compliance 

OIT Office of Information and Technology 

OMB Office of Management and Budget 

PM Project Manager 

PMAS Project Management Accountability System 

SAM Strategic Asset Management 

To Report Suspected Wrongdoing in VA Programs and Operations:
 

Telephone: 1-800-488-8244
 

E-Mail: vaoighotline@va.gov
 

(Hotline Information: http://www.va.gov/oig/contacts/hotline.asp)
 
Report Highlights: Audit of VA’s 

Project Management Accountability 

System Implementation 

Why We Did This Audit 

VA has a history of problems managing its 

information technology (IT) development 

projects. On June 19, 2009, VA launched 

the Project Management Accountability 

System (PMAS) to improve its IT 

development success rate. At the request of 

the Chief Information Officer, we conducted 

this audit to evaluate the effectiveness of 

PMAS planning and implementation. 

What We Found 

The Office of Information and Technology 

has made progress establishing PMAS. It 

has published a PMAS Guide, developed a 

prototype system for monitoring project 

status, and used the oversight approach to 

better meet incremental deliverable due 

dates for all active IT development projects. 

However, a great deal of work remains 

before PMAS can be considered completely 

established and fully operational. The 

Office of Information and Technology 

created and instituted the PMAS concept 

without a roadmap, adequate leadership, and 

staff to effectively implement and manage 

the new methodology. If such foundational 

elements are not fully implemented, the 

discipline and accountability needed for 

effective management and oversight of IT 

development projects will not be instilled. 

Specifically, key management controls to 

ensure PMAS data reliability, verify project 

compliance, and track project costs have not 

been well established. Detailed guidance on 

how such controls will be used within the 

framework of PMAS to manage and oversee 

IT projects also has not been put in place. 

Until these deficiencies are addressed, VA 

will not fully achieve the goal of PMAS to 

reduce cost overruns, schedule slippages, 

and poor performance. 

What We Recommended 

We recommended the Assistant Secretary 

for Information and Technology develop an 

implementation plan and assign adequate 

leadership and staff needed to fully execute 

PMAS. We also recommended the 

Assistant Secretary establish controls for 

ensuring data reliability, verifying project 

compliance, and tracking costs to strengthen 

PMAS oversight. Finally, we recommended 

the Assistant Secretary provide detailed 

guidance on using PMAS to ensure IT 

project success. 

Agency Comments 

The Assistant Secretary for Information and 

Technology agreed with our findings and 

recommendations and will create a detailed 

implementation plan to address the 

recommendations. We will assess and 

monitor the implementation of corrective 

actions. 

BELINDA J. FINN
 

Assistant Inspector General
 

for Audits and Evaluations
 

i 
TABLE OF CONTENTS
 

Introduction......................................................................................................................................1
 

Results and Recommendations ........................................................................................................2
 

Finding Office of Information and Technology Can Improve PMAS
 

Implementation ...................................................................................................2
 

Appendix A Background ...................................................................................................... 14
 

Appendix B Scope and Methodology................................................................................... 16
 

Appendix C Excerpt of OIT’s PMAS Dashboard Active Projects Report........................... 18
 

Appendix D Assistant Secretary for Information and Technology Comments.................... 19
 

Appendix E Office of Inspector General Contact and Staff Acknowledgements................ 22
 

Appendix F Report Distribution .......................................................................................... 23
 

ii 
Audit of VA’s Project Management Accountability System Implementation 

Objective 

Overview 

Related Prior 

Inspector 

General Report 

INTRODUCTION 

We conducted this audit to determine whether the Office of Information and 

Technology (OIT) has effectively planned and implemented the Project 

Management Accountability System (PMAS) with the management controls 

needed to ensure project managers deliver information technology (IT) 

projects within cost, schedule, and performance objectives. During meetings 

in preparation for this audit, the Assistant Secretary for Information and 

Technology expressed concerns about data reliability and the overall 

effectiveness of PMAS to control IT development projects. This audit report 

provides information to address his concerns. Appendix A provides 

background information on the PMAS initiative, while Appendix B provides 

a detailed description of our audit scope and methodology. 

VA has a history of problems managing IT development projects. For 

example, after 6 years and despite spending more than $249 million, VA 

halted the Core Financial and Logistics System (CoreFLS) project in 

2004 due to significant project management weaknesses. VA began work on 

the Financial and Logistics Integrated Technology Enterprise (FLITE) 

program in September 2005 to meet an ongoing need to address a material 

weakness in its financial management operations. 

In September 2009, we reported program managers had not fully integrated 

CoreFLS lessons learned into FLITE development (Audit of FLITE Program 

Management’s Implementation of Lessons Learned, Report 

No. 09-01467-216, September 16, 2009). In July 2010, VA canceled FLITE, 

with the exception of the Strategic Asset Management project, partly because 

FLITE had suffered from the same project management issues that plagued 

CoreFLS. VA has also experienced other project failures, such as the 

Replacement Scheduling Application project, at an estimated cost of 

$215 million. 

The Assistant Secretary for Information and Technology (the Chief 

Information Officer) proposed using PMAS, a new IT project management 

methodology, to stop the succession of IT development failures. On 

June 19, 2009, the Secretary of Veterans Affairs announced the Department 

would begin implementing PMAS to improve its IT development projects. 

In September 2009, we concluded that OIT had not established the 

management controls needed to effectively control and monitor VA’s IT 

development projects (Audit of VA’s System Development Life Cycle 

Process, Report No. 09-01239-232, September 30, 2009). We reported OIT 

did not conduct timely or thorough IT development project reviews. We also 

reported OIT did not maintain cumulative project or program life cycle cost 

and schedule documentation. OIT created PMAS in part to address these 

same issues. 

VA Office of Inspector General 1 
Audit of VA’s Project Management Accountability System Implementation 

RESULTS AND RECOMMENDATIONS 

Finding	 Office of Information and Technology Can Improve 

PMAS Implementation 

OIT has made progress in establishing PMAS. It published the PMAS 

Guide, developed a prototype project status monitoring system, instituted an 

alert process to call attention to issues that could negatively impact IT 

projects, and now manages all IT development projects under PMAS. To 

date, OIT reports that PMAS has helped meet 75 percent of the incremental 

deliverable due dates for all active IT development projects. VA’s Assistant 

Secretary for Information and Technology expects PMAS to serve as a 

model in the Federal community for managing IT investments. 

However, a great deal of work remains before PMAS can be considered 

completely established and fully operational. OIT rolled out the PMAS 

concept without a detailed plan and the leadership and staffing resources 

needed to implement the new oversight methodology. If OIT does not fully 

implement foundational elements, the discipline and accountability needed 

for effective management and oversight of VA’s IT development projects 

will not be instilled. Specifically, OIT has not established key management 

controls over data reliability, project compliance, and project costs—controls 

needed to make PMAS a viable IT oversight mechanism. Detailed 

instructions on applying PMAS to manage and oversee IT projects also are 

not in place. Until OIT addresses these deficiencies and fully implements 

PMAS, VA’s portfolio of IT development projects will remain susceptible to 

cost overruns, schedule slippages, and not meeting requirements. 

OIT Has Made OIT has taken a number of positive steps in instituting PMAS to improve 

Progress VA’s management of IT development projects. Since its inception in Implementing 

PMAS	 June 2009, the program has progressed from a concept to deployment of a 

process and automated system for managing the Department’s portfolio of IT 

development projects. With this accomplishment, PMAS is ensuring 

incremental deliverables and shows promise of serving as a model oversight 

methodology for wider Federal use. 

PMAS Concept PMAS represents a major shift from the way VA historically has planned 

and managed IT development projects. PMAS was designed as a 

performance-based management discipline that provides incremental 

delivery of IT system functionality—tested and accepted by customers— 

within established schedule and cost criteria. The PMAS concept requires 

projects delivering functional business capability in cycles of 6 months or 

less. PMAS is used to plan and manage all IT development projects that 

introduce new functionality or enhance existing capabilities within current 

VA systems that cost more than $250,000 in one fiscal year. PMAS does not 

apply to projects associated with the sustainment of existing IT systems. 

VA Office of Inspector General 2 
Audit of VA’s Project Management Accountability System Implementation 

A project manager (PM) and an integrated project team (IPT) are responsible 

for managing each project subject to PMAS oversight. The PM is primarily 

responsible for delivering expected outcomes within cost, schedule, and 

scope. The IPT, co-chaired by the business sponsor and the PM, serves as 

the governing mechanism with responsibility for overall project decisions. 

IPTs comprise the business sponsor, significant project stakeholders, OIT 

representatives, and support staff from the Office of General Counsel and the 

Office of Acquisition and Logistics. 

Each month, PMs enter updated data on the status of their projects into a 

PMAS database, which OIT uses for centralized oversight. Projects under 

PMAS are categorized into one of five states—new start, planning, active, 

paused, and closed. The level of monitoring done on a project is dependent 

on the state it holds in the project management life cycle. For example, a 

project in the paused state is much more heavily scrutinized than a project in 

the new start state. 

During the planning state, PMs and IPTs ensure that required documents 

such as requirements specification documents, system design documents, and 

project management plans are developed to demonstrate that the projects are 

ready to enter the active state. To receive the Assistant Secretary for 

Information and Technology’s approval to become active, a project must 

have: 

	 A business sponsor 

	 An established IPT 

	 A plan for delivering incremental functionality to the intended 

customer at least every 6 months 

	 Documented requirements for initial incremental deliverables 

	 Success criteria established and jointly accepted by the business 

sponsor, OIT, and the vendor 

Active projects involve processes to build and deliver incremental 

functionality according to the established project schedule, or at least every 

6 months. Customers certify incremental delivery of functionality on a 

customer acceptance form. The PM must also sign the customer acceptance 

form, validating that the incremental requirements have been met. Three 

failures to deliver functionality as scheduled should result in a project being 

paused. Once a project has been paused, no further development can occur 

until it has been evaluated, re-planned, and approved to restart by the 

Assistant Secretary for Information and Technology. The alternative is to 

terminate the project altogether. 

VA Office of Inspector General 3 
Audit of VA’s Project Management Accountability System Implementation 

PMAS 

Implementation 

Progress 

Since instituting PMAS, OIT has made the following progress in its 

development and implementation. 

	 Published the PMAS Guide, which is the official OIT policy for 

managing IT development projects. In broad terms, it provides the 

procedures for planning, monitoring, and managing the projects. It 

presents an overview of PMAS processes, roles and responsibilities, 

and management controls. OIT published the initial guide on 

March 29, 2010, and the first revision on September 17, 2010. 

	 Developed a prototype PMAS Dashboard that displays the 

information contained in the PMAS database. OIT officials use the 

dashboard to monitor IT development project progress and identify 

issues with performance. The dashboard provides color-coded 

ratings of actual project performance against cost, schedule, and 

performance targets. A green rating indicates that a project is on 

track, a yellow that a project is behind, and a red that a project is 

failing. The prototype PMAS Dashboard became operational in May 

2010. OIT expects to have the final PMAS Dashboard in place by 

September 2013. 

	 Created reports from the PMAS Dashboard to facilitate oversight of 

VA’s IT development projects. The Assistant Secretary for 

Information and Technology and other OIT senior leadership use the 

Daily PMAS Report and related products to determine how well the 

Product Development organization is performing in meeting its 

incremental delivery schedules. Appendix C provides an excerpt of 

the PMAS Dashboard Active Projects Report. 

	 Established the red flag process, which allows anyone associated with 

a project to elevate issues impacting the project to senior level 

officials in a timely manner so they can take corrective actions 

quickly. Project managers have used this process successfully to 

obtain needed personnel resources for projects. 

	 Converted all ongoing and new product development projects to 

PMAS. 

As of May 2011, OIT was managing 119 active IT development projects 

using PMAS. An additional 60 projects were in the planning state, while 

41 projects were classified as new starts. At the time of our audit, OIT’s 

efforts were primarily focused on ensuring that IT development projects 

were achieving their incremental deliveries. On May 11, 2011, in a 

statement before the House Committee on Veterans’ Affairs, the Assistant 

Secretary for Information and Technology said that, through PMAS, OIT had 

achieved approximately 75 percent of the incremental delivery milestone 

dates for all active IT development projects. 

VA Office of Inspector General 4 
Audit of VA’s Project Management Accountability System Implementation 

Lack of a Plan 

and Personnel 

To Lead 

No 

Implementation 

Plan 

Further, according to the Assistant Secretary for Information and 

Technology, OIT has made substantial strides forward and is well on its way 

toward achieving the goal of being the best IT organization in the Federal 

Government. He said that in a number of areas, including PMAS, VA has 

blazed a trail of innovation that the rest of Government is beginning to 

follow. 

While these initial accomplishments are encouraging, the current framework 

for PMAS does not provide a sufficient foundation for continued progress. 

Because PMAS is a new management framework created at VA, OIT 

proceeded with PMAS’ implementation without the benefit of a roadmap— 

that is, a detailed implementation plan—to guide it or the leadership and 

staffing resources needed to support it through full implementation. 

When the Secretary of Veterans Affairs announced PMAS on June 19, 2009, 

it was in the preliminary stages of development. It was primarily the 

Assistant Secretary for Information and Technology’s vision rather than a 

fully developed methodology for managing IT development projects. The 

Assistant Secretary tasked OIT with putting his vision for PMAS into 

practice without providing a detailed plan to guide it through 

implementation. As such, OIT launched PMAS before it was well defined, 

and before it had in place the structure and support needed to effectively 

institute the new methodology. 

An implementation plan would have identified the tasks necessary to 

accomplish PMAS, who would complete them, target due dates, and how the 

work would be tracked and monitored. OIT’s FY 2010 Operating Plan 

included a general outline for implementing PMAS, but it lacked details. 

According to the Operating Plan, the Principal Deputy Assistant Secretary 

for Information and Technology was responsible for developing a plan of 

action with milestones by December 2009, but this was not done. OIT 

personnel responsible for implementing PMAS told us that they primarily 

relied on and reacted to directions provided by the Assistant Secretary for 

Information and Technology to implement PMAS. 

Along with milestones and processes for managing PMAS, a detailed 

implementation plan would have also included performance measures to 

assess how well PMAS is meeting OIT’s goals for IT development projects. 

Performance measures are indicators or metrics used to gauge program 

success—whether a program is achieving its objectives and whether progress 

is being made toward attaining organizational goals. Such goals are the 

target levels of performance expressed as measureable objectives against 

which actual achievement can be compared. To achieve optimal 

effectiveness with PMAS, OIT needs to develop a mix of both short- and 

long-term goals that address performance targets several years into the 

future. 

VA Office of Inspector General 5 
Audit of VA’s Project Management Accountability System Implementation 

Inadequate 

Leadership and 

Staff 

Management 

Controls Not 

Established 

Unreliable Data 

OIT’s efforts to make PMAS fully operational are further hindered because it 

proceeded with the implementation even though it did not have the 

leadership and staffing resources in place that were needed to establish the 

new process. Although OIT’s FY 2010 Operating Plan included provisions 

for implementing a PMAS organizational structure, OIT did not adhere to the 

plan. According to the Operating Plan, the Principal Deputy Assistant 

Secretary for Information and Technology was responsible for establishing a 

PMAS office by May 2010. The Director of the PMAS Office would be 

responsible for: 

	 Developing and implementing the PMAS process 

	 Monitoring and controlling all IT development projects, including 

initiating reviews of the projects by September 2010 

	 Managing an office of analysts and communications personnel to 

monitor all IT development projects to ensure compliance with 

PMAS processes and milestones 

However, OIT did not appoint a PMAS Director or establish a PMAS Office. 

At the time of our audit, OIT had less than two full-time personnel dedicated 

to the implementation of PMAS—a PM who was primarily responsible for 

developing the PMAS Dashboard and a consultant whose duties included 

supporting the program. At various points in time, OIT assigned additional 

personnel to work on PMAS tasks, but on a part-time or voluntary basis. For 

example, in the absence of a PMAS Office, the Director of the Office of 

Program Management, Policy, Assessment, and Reporting was tasked to 

create the PMAS Guide. 

In October 2010, OIT completed a reorganization intended to transform it to 

meet the Secretary’s vision for a 21st Century VA. In lieu of a central PMAS 

Office, pockets of PMAS responsibilities were created throughout the OIT 

organization; no single office or group of individuals was designated to 

orchestrate all of the actions needed to roll out the new IT oversight process. 

OIT also did not clearly define PMAS roles and responsibilities. As a result, 

even the PM and consultant (the two individuals most familiar with PMAS) 

told us that they had trouble identifying the personnel responsible for the 

PMAS tasks that needed to be accomplished. 

OIT has not yet established key management controls needed to make PMAS 

a sound IT oversight mechanism. Lacking a detailed implementation plan 

and the leadership and program staff needed to guide efforts, OIT may face 

challenges making PMAS fully operational and providing the accountability 

and discipline required to effectively manage all IT developments. 

The PMAS Dashboard was designed to provide the project status 

information needed for OIT leaders to make timely and informed decisions 

on IT development initiatives. However, we found that the information 

VA Office of Inspector General 6 
Audit of VA’s Project Management Accountability System Implementation 

maintained on the PMAS Dashboard does not always reflect the actual 

performance of an IT development project. 

Office of Management and Budget (OMB) Circular A-11, Part 7, 

Section 300, Planning, Budgeting, Acquisition, and Management of Capital 

Assets, provides guidance and reporting requirements for major IT 

development projects. It requires Federal agencies to institute performance 

measures and management processes for monitoring and comparing planned 

results against actual performance. OMB expects agencies to achieve, on 

average, 90 percent of cost, schedule, and performance goals. An agency 

must review all projects not achieving 90 percent of its goals to determine 

whether it still needs the projects and what corrective actions, including 

possible termination, should be taken. In addition, OMB’s Information 

Technology Investment Baseline Management Policy (M-10-27), issued 

June 28, 2010, requires agencies to measure performance against both the 

current approved baseline and the original baseline. 

In line with these requirements, the PMAS Guide states that IT project 

progress should be primarily measured by comparing actual performance 

data against a project’s baseline. When performance deviates significantly 

from the plan, management must take timely corrective actions. 

However, OIT does not maintain in the PMAS Dashboard an audit trail of 

the initial baseline against which to measure project progress. Consequently, 

OIT has limited assurance that the information maintained in the system is 

accurate and complete. Currently, when a PM enters a revised project 

delivery date into the PMAS Dashboard, OIT loses visibility of the original 

delivery date. The current Dashboard update procedure simply entails 

replacing existing project delivery information with a revised due date. 

Compounding this issue is the fact that OIT has not yet begun performing the 

independent reviews needed to validate the accuracy of the data PMs enter 

into the PMAS Dashboard on their projects. As a result, OIT cannot rely on 

the PMAS Dashboard to support project oversight and provide accurate 

project status data OIT leaders need to make timely and informed 

management decisions. 

We compared the June 28, 2010, Daily PMAS Report with the 

October 19, 2010, Daily PMAS Report to determine whether projects 

successfully completed 50 incremental deliveries scheduled to occur during 

September 2010. We found that 13 of the 50 incremental deliveries had not 

been completed as originally scheduled. Although OIT officials had 

approved each of the schedule slippages, they lost visibility of the baseline 

schedules when PMs entered revised due dates into the PMAS Dashboard. 

The BusinessWare Migration project provides a useful illustration of this 

deficiency. The project was scheduled to complete an initial operating 

capability test on September 28, 2010. 

VA Office of Inspector General 7 
Audit of VA’s Project Management Accountability System Implementation 

OIT officials extended this incremental delivery upon approving a revised 

completion date of November 30, 2010. As soon as the PM entered the 

revised date into the PMAS Dashboard, OIT lost visibility of the original 

baseline information. Consequently, although the project schedule had 

slipped by 2 months, the PMAS Dashboard indicated that the project was on 

track. 

The project subsequently missed the revised completion date. It received a 

strike during a meeting with OIT officials held on January 18, 2011. During 

that meeting, OIT officials again agreed to extend the test completion date to 

March 30, 2011. The project received another strike when it did not meet the 

latest revised completion date. However, as of April 22, 2011, the PMAS 

Dashboard showed no indication that the project was at risk. Not only did 

the PMAS Dashboard incorrectly indicate that the project was on schedule, it 

also indicated that the project had not received any strikes. In May 2011, the 

PMAS Dashboard was updated to show that the project had three strikes 

against it and was still ongoing and meeting schedule even though the dates 

had changed several times. 

The Strategic Asset Management (SAM) project provides another example 

of the limited usefulness of the information captured on the PMAS 

Dashboard. On April 21, 2009, VA awarded a task order for the SAM 

project valued at approximately $8 million. The period of performance for 

the task order was 12 months. Modifications increased the value of the task 

order to over $20 million, more than doubling the value of the task order and 

the period of performance. After the project received its third strike for 

failing to meet a delivery milestone, the Assistant Secretary for Information 

and Technology suspended the SAM project in February 2011, just weeks 

before it was scheduled to go live. 

Despite SAM’s troubled history, the PMAS Dashboard continued to show all 

facets of the project in a green status, signifying that the project was on track 

right up until the day it was suspended. In addition, the PMAS Dashboard 

indicated that the project had not received a single strike when in reality it 

had already received two strikes prior to suspension. The PMAS Dashboard 

also did not reflect that the project had incurred significant cost overruns and 

schedule slippages, demonstrating that the Dashboard needs improvements to 

become an effective and reliable tool for project monitoring. 

OIT will not be able to realize the intended benefits of the PMAS Dashboard 

until it establishes the procedures and controls needed to ensure that the data 

presented by the dashboard are reliable and provide a complete picture of 

each project’s status. 

VA Office of Inspector General 8 
Audit of VA’s Project Management Accountability System Implementation 

Planning, 

Compliance, 

and Outcome 

Reviews Not 

Performed 

OIT has not conducted reviews required under PMAS to validate the 

performance of IT development projects. According to the PMAS Guide, 

these reviews are to examine project planning, compliance, and outcomes. 

Specifically, as of September 17, 2010, the PMAS Guide required that OIT 

review projects in the planning state every 60 calendar days to evaluate 

whether they should be moved to the active state or closed. Such reviews are 

critical to ensure projects do not remain in planning for an excessive amount 

of time, thus expending unnecessary resources without making sufficient 

progress. To evaluate compliance with this requirement, we asked OIT 

officials to provide documentation of completed reviews for 59 IT 

development projects. The projects were identified on the PMAS Dashboard 

as being in the planning state as of February 3, 2011. OIT officials were 

unable to provide evidence to show that they had done any of the required 

planning reviews. OIT officials stated that their management oversight had 

suffered because of insufficient staff resources. 

Further, OIT did not conduct compliance and outcome reviews. According 

to the PMAS Guide, compliance reviews are independent periodic reviews 

intended to ensure projects comply with PMAS requirements. In 

comparison, outcome reviews are to ensure that incremental deliverables 

meet customer expectations, and that projects provide expected functionality 

within designated time and budget parameters. These reviews are critical for 

ensuring data integrity since PMAS Dashboard data are self-reported by 

PMs. All of the PMs in our case studies of five IT development projects said 

that their projects had not undergone such reviews. Responsible OIT 

officials confirmed that OIT was not conducting the reviews because they 

had not yet developed detailed review processes and staffing resources were 

insufficient to implement an adequate independent review function. 

The Assistant Secretary for Information and Technology has periodically 

required PMs to provide status briefings to support his review of selected IT 

development projects. In addition, the Assistant Secretary has held strike 

meetings to discuss why certain projects failed to meet scheduled 

incremental deliveries, as well as red flag meetings to resolve risks and 

issues negatively impacting projects. However, such briefings and meetings 

have typically been conducted on an ad hoc basis and do not constitute a 

consistent means of reviewing the performance of IT development projects. 

On March 28, 2011, the Office of Information and Technology Oversight 

and Compliance (ITOC) began the first round of PMAS compliance reviews. 

ITOC staff developed standard operating procedures that identified roles and 

responsibilities, processes, and reporting guidelines to aid this review effort. 

ITOC officials acknowledged that their personnel needed training to develop 

the skills needed to perform these reviews. They explained that this was a 

new mission for their organization. They also stated that due to time 

constraints, initial training would be limited in depth; as such, initial 

VA Office of Inspector General 9 
Audit of VA’s Project Management Accountability System Implementation 

compliance reviews would also be limited in depth. ITOC officials 

anticipate increasing the extent of the reviews over time as ITOC personnel 

receive more training and validate the new review procedures. 

Lack of Project 

Budget and 

Cost Tracking 

Lack of Detailed 

Guidance 

OIT has not established budget traceability down to project and increment 

levels within the OIT Operating Plan. According to the PMAS Guide, a 

clearly established and understandable line of sight should exist from OMB 

Exhibit 300s, to programs, to projects, and finally to PMAS incremental 

deliveries. Exhibit 300s are the documents OMB uses to collect IT project 

investment information from each agency. Full traceability from this budget 

document to expenditures is necessary to ensure decision makers have 

sufficient information to understand the potential impact of decisions made 

on IT development projects. 

However, responsible OIT officials stated that current VA financial systems 

only allow them to track obligations for each project, but not actual costs 

incurred. OIT has not developed a standard methodology PMs can use to 

overcome this problem and ensure compliance with the traceability 

requirements in the PMAS Guide. Without accurate expenditure data, OIT 

cannot effectively compare planned against actual performance to determine 

whether VA is meeting OMB’s mandate to achieve 90 percent of its cost 

goals for IT development projects. 

In June 2009, OIT and project personnel were required to implement PMAS 

before detailed guidance was provided on the processes, deliverables, and 

management controls it would entail. Initially, on July 9, 2009, OIT 

published PMAS Instructions for Project Managers, which targeted the 

initial projects managed by PMAS—45 projects paused by the Assistant 

Secretary for Information and Technology as a result of an OIT analysis of 

VA’s IT projects. The intent of this document was to inform PMs of the 

steps they needed to take to start managing paused projects, along with the 

actions needed to gain approval to restart the projects under PMAS. 

However, the document fell short in several areas. For example, it did not 

provide a description of PMAS and included no discussion of PMAS 

processes because they had not yet been developed by OIT. It also did not 

explain how to manage a project under PMAS, how OIT would monitor 

projects, and the roles and responsibilities for personnel responsible for 

managing and monitoring projects. 

As previously stated, the PMAS Guide was ultimately published in 

March 2010, approximately 9 months after PMAS was announced. The 

PMAS Guide provided an overview of PMAS, described the PMAS 

processes, and defined critical terms such as incremental deliverables. It 

identified management controls for monitoring performance and ensuring 

that PMAS procedures were followed. It also explained management and 

oversight roles and responsibilities. OIT used the PMAS Guide to introduce 

VA Office of Inspector General 10 
Audit of VA’s Project Management Accountability System Implementation 

new PMAS concepts and inject more accountability and discipline into the 

new methodology. For example, the PMAS Guide established that three 

failures (strikes) to meet scheduled delivery of functionality would result in a 

project being paused. Nonetheless, this guide, and the revision in 

September 2010 , still did not provide all of the step-by-step instructions 

needed on how to comply with PMAS requirements. 

The lack of detailed guidance resulted in confusion on issues such as what 

constituted an acceptable deliverable to meet the 6-month delivery 

requirement. For example, the PM for the Veterans Health Information and 

Technology Architecture Imaging Storage project originally planned only 

one incremental delivery—national implementation of a patch to provide 

functionality needed to maintain system integrity. The PM scheduled the 

incremental delivery merely to meet the PMAS 6-month delivery 

requirement. He also scheduled it without a complete understanding of what 

constituted an acceptable PMAS deliverable and a project plan to facilitate 

meeting the delivery schedule. The PM was unaware that testing of the 

patch, which should have occurred before patch implementation, constituted 

an acceptable PMAS incremental delivery. After recognizing his error, the 

PM revised the project delivery schedule so that the testing of the patch 

would comprise the first deliverable, and national deployment of the patch 

would comprise the second. 

The project has been in active state since October 2009; however, the first 

incremental delivery—testing of the software patch—was not delivered until 

November 8, 2010. Delivery 2, national implementation of the patch, was 

originally scheduled to be completed on March 31, 2010. OIT officials once 

more extended the second incremental delivery 14 months by approving a 

revised software patch implementation date of June 3, 2011, which the 

project met. Because of the lack of initial PMAS oversight, OIT allowed the 

project to remain in an active state without penalty even though it was not 

producing incremental deliveries of functionality every 6 months as required. 

To further complicate matters, OIT has not developed written processes and 

procedures that describe how management controls, such as independent 

reviews, budget traceability, and planning reviews, will be implemented; 

what they are to accomplish; and who is responsible for carrying out the 

tasks associated with the management controls. Although the PMAS Guide 

provides general guidance in these areas, it does not provide sufficient 

detailed instructions to effectively implement these needed management 

controls. 

For example, the PMAS Guide states that OIT will monitor a project’s 

progress by comparing actual performance data against the project’s cost and 

schedule baseline. Although OIT officials acknowledged that VA’s financial 

systems do not facilitate the tracking of costs for each project, they have not 

developed standard written procedures that tell PMs how to accumulate and 

VA Office of Inspector General 11 
Conclusion 

Audit of VA’s Project Management Accountability System Implementation 

track project costs and how to develop an audit trail of the costs for 

independent verification. Similarly, the PMAS Guide does not elaborate on 

essential details such as the frequency and scope of compliance, outcome, 

and planning reviews; the methodology for the reviews; consequences of 

unfavorable reviews; and reporting requirements. Senior OIT officials 

agreed that they needed to develop detailed policies and procedures to guide 

OIT and project personnel through these PMAS processes. 

According to the PMAS Guide, PMAS processes were designed to enable 

leadership and project management to clearly see cost, schedule, and quality 

status. However, OIT efforts to date have focused primarily on whether 

projects are meeting scheduled incremental deliveries of functionality. Until 

OIT fully defines and documents the processes and procedures for carrying 

out the management controls mentioned above, they will not achieve the 

intended accountability and discipline needed to make PMAS more 

successful. 

OIT has an opportunity to set the Federal standard for IT project 

management. However, the current PMAS framework does not provide a 

sound basis for future success. A great deal of work remains before PMAS 

can be considered completely established and fully operational. Currently, 

the PMAS Dashboard’s usefulness as a project management and 

performance monitoring tool is limited. The information maintained on the 

dashboard is not consistently reliable and does not provide the project 

performance history needed to help senior VA leaders make informed project 

decisions. Further, designating less than two full-time personnel to 

implement and manage PMAS to monitor VA’s entire portfolio of IT 

development projects has proven a questionable approach, as evidenced by 

the finding in this report. 

The initial steps accomplished to implement PMAS have been critical to 

improve VA’s IT project management and oversight. However, it remains 

too early to assess PMAS’ impact—most steps have recently become 

operational or are not fully implemented. Additionally, only recently have 

IT projects been required to comply and incrementally deliver under 

PMAS—most have gone through only one or two incremental delivery 

milestones since PMAS’ inception in June 2009. Thus, the overall 

effectiveness of PMAS in improving VA’s IT development project success 

cannot be clearly established. 

OIT’s ability to complete PMAS implementation in an effective and timely 

manner would be significantly aided by a well thought out implementation 

plan, with details on the controls needed for viable PMAS oversight and IT 

project accountability, as well as additional program leadership and staff 

resources. Putting these foundational elements in place to provide stronger 

visibility and oversight of VA’s portfolio of IT development projects will 

VA Office of Inspector General 12 
Audit of VA’s Project Management Accountability System Implementation 

Recommendations 

Management 

Comments and 

OIG Response 

help reduce the risk of cost overruns, schedule slippages, and poor 

performance. 

1.	 We recommended the Assistant Secretary for Information and 

Technology develop a detailed plan of the steps needed to complete 

implementation of the Project Management Accountability System 

program, including milestones for deliverables, performance measures, 

and a methodology for tracking progress. 

2.	 We recommended the Assistant Secretary for Information and 

Technology assess Project Management Accountability System resources 

to ensure the Office of Information and Technology has adequate 

leadership and staff assigned to complete both implementation and 

execution of the Project Management Accountability System. 

3.	 We recommended the Assistant Secretary for Information and 

Technology clearly define the roles and responsibilities for executing the 

Project Management Accountability System and establish a central office 

or group of individuals responsible for fully implementing and executing 

the Project Management Accountability System. 

4.	 We recommended the Assistant Secretary for Information and 

Technology modify the Project Management Accountability System 

Dashboard to maintain original baseline data and issue guidance to 

ensure project performance is measured against both the original and 

current baselines. 

5.	 We recommended the Assistant Secretary for Information and 

Technology designate personnel and provide them with detailed written 

procedures to perform periodic independent reviews of the Project 

Management Accountability System Dashboard to ensure data reliability 

and completeness. 

6.	 We recommended the Assistant Secretary for Information and 

Technology designate personnel and provide them with detailed written 

instructions to perform Project Management Accountability System 

planning, outcome, and compliance reviews and track project budgets 

and costs. 

The Assistant Secretary agreed with our findings and recommendations. OIT 

will create an implementation plan within 60 days of the report issuance date 

addressing in detail how it will implement each recommendation in the audit 

report. We will assess these actions when we receive the details and follow 

up on the implementation of corrective actions. Appendix D contains the full 

text of the comments received from the Assistant Secretary for Information 

and Technology. 

VA Office of Inspector General 13 
Audit of VA’s Project Management Accountability System Implementation 

Appendix A 

Overview of 

PMAS 

PMAS 

Principles 

Roles and 

Responsibilities 

Background 

The Secretary of Veterans Affairs recently identified IT as a critical resource 

for achieving the President’s vision of a 21st Century Department of Veterans 

Affairs. In particular, the Secretary recognized the need for VA to leverage 

the power of IT to accelerate and modernize the delivery of benefits and 

services to our nation’s veterans. 

However, VA historically has struggled to manage IT development projects 

that successfully deliver desired results within cost, schedule, and 

performance objectives. To improve, OIT established PMAS in 2009, 

signifying a substantial change in the way VA plans and manages IT 

development projects. PMAS was designed to reduce risks; to institute 

monitoring, controlling, and reporting discipline; and to establish 

accountability. It became mandatory that VA personnel use PMAS to plan 

and manage all IT development projects that introduce new functionality or 

enhance existing capabilities within current VA systems. 

Concurrent with PMAS’s implementation, OIT’s Transformation 21 IT 

Work Group conducted an initial analysis of 282 of VA’s IT development 

projects to support the President’s vision for creating a 21st Century VA. As 

a result of the Work Group’s analysis, the Assistant Secretary for 

Information and Technology paused 45 of the projects because they were 

either 1 year behind schedule or 50 percent over their baseline budgets. 

These 45 projects were the first projects to be re-planned and managed under 

PMAS and were required to meet the new incremental delivery requirements. 

OIT ultimately canceled 12 of the 45 projects, while it managed the 

33 remaining projects using PMAS. 

When fully implemented, PMAS will be a performance-based management 

discipline that provides frequent delivery of deployable IT system 

functionality—tested and accepted by customers—within established 

schedule and cost criteria. Throughout project execution, customers will 

certify delivery of IT functionality at intervals of 6 months or less. Three 

failures to meet a delivery of functionality will result in a project being 

paused or suspended. Once a project has been paused, no further 

development activity can occur until it has been evaluated for cause, replanned, and approved to restart. Alternatively, a paused project might be 

terminated. The Assistant Secretary for Information and Technology or his 

designee is responsible for approving project restarts. 

The Assistant Secretary for Information and Technology is responsible for 

monitoring all IT development projects operating under PMAS. PMs are 

primarily responsible for managing IT development projects to deliver 

expected outcomes on time and within budget. Accordingly, under PMAS, 

VA Office of Inspector General 14 
Audit of VA’s Project Management Accountability System Implementation 

they are responsible for ensuring that their projects deliver IT functionality in 

increments of 6 months or less and that their projects comply with cost and 

schedule milestones. PMs also work as part of IPTs that serve as core 

multidisciplinary management groups responsible for the success of their 

projects. The PM and the business sponsor co-chair the IPT. IPTs typically 

include team members from the Offices of Acquisition and Logistics and 

General Counsel, in addition to various IT personnel. 

VA Office of Inspector General 15 
Audit of VA’s Project Management Accountability System Implementation 

Appendix B 

Scope 

Methodology 

Reliability of 

ComputerProcessed 

Data 

Scope and Methodology 

We conducted our audit work from July 2010 through June 2011. We 

focused on evaluating the actions OIT has taken to plan and implement 

PMAS and the progress it has made in this undertaking. We also evaluated 

whether OIT has established management controls to ensure that project 

managers deliver IT development projects that meet cost, schedule, and 

performance objectives. As of May 2011, OIT was managing 119 active IT 

development projects using PMAS. An additional 60 projects were in the 

planning state, while 41 projects were classified as new starts. 

We reviewed OMB and VA guidance related to effective management and 

oversight of IT development projects. We also reviewed PMAS guidance 

developed by OIT to provide criteria for our audit. To evaluate whether 

OIT had effectively planned the implementation of PMAS, we interviewed 

the Assistant Secretary for Information and Technology and other senior 

OIT officials, as well as PMAS and project management officials. In 

addition, we interviewed senior-level acquisition officials and the 

contracting officers for several IT development projects. We also examined 

the steps OIT had taken to implement PMAS. 

To evaluate whether OIT has established the management controls needed 

to ensure project managers deliver projects that meet cost, schedule, and 

performance objectives, we interviewed senior OIT leaders and analyzed 

PMAS guidance and processes. We interviewed PMs and customers and 

reviewed project documentation to analyze whether internal controls are 

effective and whether PMAS is operating as intended. We also selected and 

analyzed IT development case studies to evaluate whether PMs and OIT 

personnel are complying with PMAS requirements. 

In our effort to determine whether OIT effectively planned and 

implemented PMAS, we reviewed the extent and quality of management 

controls over the data contained in the PMAS Dashboard. We interviewed 

OIT and project management officials and collected and reviewed 

documentation to gain an understanding of the existing management 

controls. We evaluated whether the controls in place were adequate to 

provide reasonable assurance that the data in the dashboard were valid and 

reliable for our intended use. We also reviewed the data in the dashboard 

for selected projects to determine whether the data provided a valid and 

reliable representation of project status. 

We concluded that the data were not sufficiently reliable to determine the 

actual performance of IT development projects. As a result, we developed 

recommendations for improving management controls to ensure the 

reliability of data included in the PMAS Dashboard. 

VA Office of Inspector General 16 
Audit of VA’s Project Management Accountability System Implementation 

Compliance 

With 

Government 

Audit 

Standards 

Our assessment of internal controls focused on those controls relating to our 

audit objectives. We conducted this performance audit in accordance with 

generally accepted government auditing standards. These standards require 

that we plan and perform the audit to obtain sufficient, appropriate evidence 

to provide a reasonable basis for our findings and conclusions based on our 

audit objective. We believe that the evidence obtained provides a 

reasonable basis for our findings and conclusions based on our audit 

objective. 

VA Office of Inspector General 17 
Audit of VA’s Project Management Accountability System Implementation 

Appendix C Excerpt of OIT’s PMAS Dashboard Active Projects Report
 

VA Office of Inspector General 18 
Audit of VA’s Project Management Accountability System Implementation 

Appendix D	 Assistant Secretary for Information and Technology 

Comments 

Department of Memorandum Veterans Affairs 

Date:	 August 1, 2011 

From:	 Assistant Secretary for Information and Technology (005) 

Subj:	 Draft Report-Audit of VA’s Implementation of the Project Management 

Accountability System; Project No. 2010-03162-R6-0328 

To: Assistant Inspector General for Audits and Evaluations (52) 

Thank you for the opportunity to review the subject Office of Inspector General 

(OIG) draft report. I appreciate the work that your staff has done, and the 

contribution that their work and this report make to the continued success of the 

PMAS program. The Office of Information and Technology concurs with the 

OIG’s findings and submits the attached written comments for each of the 

recommendations. 

As you are aware, PMAS has been a key component in our ability to dramatically 

improve the results of VA’s IT investments over the last two years. At the same 

time, PMAS has been a constantly evolving program itself, as we sought to 

establish policies and disciplines that were both effective and pragmatic. In June 

of 2009, PMAS was solely a vision and a set of principles. Since that time, 

through the contributions of hundreds of VA staff, PMAS has both become a real 

proram itself. I has been the primary driver in the renovation of an IT 

organization of over 2, 500 people, and has become the accepted “way we work” 

in our Product Development organization. Most importantly, the implementation 

of PMAS has dramatically reduced the number of failing IT projects at VA, and 

saved substantial taxpayer dollars. We appreciate your assistance in helping 

define how PMAS can be improved. 

If you have any questions, contact me at (202) 461-6910, or have a member of 

your staff contact Lorraine Landfried at (202) 632-4347. 

VA Office of Inspector General 19 
Audit of VA’s Project Management Accountability System Implementation 

OIG DRAFT REPORT 

Audit of the Project Management Accountability System Implementation 

(Project No. 2010-03162-R6-0328) 

OIG Recommendations 

1. We recommend the Assistant Secretary for Information and Technology develop a 

detailed plan of the steps needed to complete implementation of the Project Management 

Accountability System program, including milestones for deliverables, performance 

measures, and a methodology for tracking progress. 

CONCUR. The Office of Information and Technology is preparing an implementation plan to 

address the audit recommendations. The implementation plan will address in detail how OIT 

will implement each OIG recommendation in the audit report. 

TARGET COMPLETION DATE: The plan will be completed 60 days after the final report 

issuance date. 

2. We recommend the Assistant Secretary for Information and Technology assess Project 

Management Accountability System resources to ensure the Office of Information and 

Technology has adequate leadership and staff assigned to complete both implementation and 

execution of the Project Management Accountability System. 

CONCUR. The Office of Information and Technology is preparing an implementation plan to 

address the audit recommendations. The implementation plan will address in detail how OIT 

will implement each OIG recommendation in the audit report. 

TARGET COMPLETION DATE: The plan will be completed 60 days after the final report 

issuance date. 

3. We recommend the Assistant Secretary for Information and Technology clearly define 

the roles and responsibilities for executing the Project Management Accountability System 

and establish a central office or group of individuals responsible for fully implementing and 

executing the Project Management Accountability System. 

CONCUR. The Office of Information and Technology is preparing an implementation plan to 

address the audit recommendations. The implementation plan will address in detail how OIT 

will implement each OIG recommendation in the audit report. 

TARGET COMPLETION DATE: The plan will be completed 60 days after the final report 

issuance date. 

4. We recommend the Assistant Secretary for Information and Technology modify the 

Project Management Accountability System Dashboard to maintain original baseline data 

VA Office of Inspector General 20 
Audit of VA’s Project Management Accountability System Implementation 

and issue guidance to ensure project performance is measured against both the original 

and current baselines. 

CONCUR. The Office of Information and Technology is preparing an implementation plan to 

address the audit recommendations. The implementation plan will address in detail how OIT 

will implement each OIG recommendation in the audit report. 

TARGET COMPLETION DATE: The plan will be completed 60 days after the final report 

issuance date. 

5. We recommend the Assistant Secretary for Information and Technology designate 

personnel and provide them with detailed written procedures to perform periodic independent 

reviews of the Project Management Accountability System Dashboard to ensure data 

reliability and completeness. 

CONCUR. The Office of Information and Technology is preparing an implementation plan to 

address the audit recommendations. The implementation plan will address in detail how OIT 

will implement each OIG recommendation in the audit report. 

TARGET COMPLETION DATE: The plan will be completed 60 days after the final report 

issuance date. 

6. We recommend the Assistant Secretary for Information and Technology designate 

personnel and provide them with detailed written instructions to perform Project 

Management Accountability System planning, outcome, and compliance reviews and track 

project budgets and costs. 

CONCUR. The Office of Information and Technology is preparing an implementation plan to 

address the audit recommendations. The implementation plan will address in detail how OIT 

will implement each OIG recommendation in the audit report. 

TARGET COMPLETION DATE: The plan will be completed 60 days after the final report 

issuance date. 

VA Office of Inspector General 21 
Audit of VA’s Project Management Accountability System Implementation 

Appendix E Office of Inspector General Contact and Staff 

Acknowledgements 

OIG Contact	 For more information about this report, please 

contact the Office of Inspector General at 

(202) 461-4720. 

Acknowledgments Mario M. Carbone, Director 

Clenes Duhon 

Michael Jacobs 

Jehri Lawson 

Theresa Lospinoso 

Kristin Nichols 

Charanpreet Singh 

VA Office of Inspector General 22 
Audit of VA’s Project Management Accountability System Implementation 

Appendix F Report Distribution 

VA Distribution 

Office of the Secretary 

Veterans Health Administration 

Veterans Benefits Administration 

National Cemetery Administration 

Assistant Secretaries 

Office of General Counsel 

Non-VA Distribution 

House Committee on Veterans’ Affairs 

House Appropriations Subcommittee on Military Construction, Veterans 

Affairs, and Related Agencies 

House Committee on Oversight and Government Reform 

Senate Committee on Veterans’ Affairs 

Senate Appropriations Subcommittee on Military Construction, Veterans 

Affairs, and Related Agencies 

Senate Committee on Homeland Security and Governmental Affairs 

National Veterans Service Organizations 

Government Accountability Office 

Office of Management and Budget 

This report will be available in the near future on the OIG’s Web site at 

http://www.va.gov/oig/publications/reports-list.asp. This report will remain 

on the OIG Web site for at least 2 fiscal years. 

VA Office of Inspector General 23