Software Development: Effective Practices and Federal Challenges in Applying Agile Methods
Published: 2012-07-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal agencies depend on IT to support their missions and spent at least $76 billion on IT in fiscal year 2011. However, long-standing congressional interest has contributed to the identification of numerous examples of lengthy IT projects that incurred cost overruns and schedule delays while contributing little to mission-related outcomes. To reduce the risk of such problems, the Office of Management and Budget (OMB) recommends modular software delivery consistent with an approach known as Agile, which calls for producing software in small, short increments. Recently, several agencies have applied Agile practices to their software projects.
Accordingly, GAO was asked to identify (1) effective practices in applying Agile for software development solutions and (2) federal challenges in implementing Agile development techniques. To do so, GAO identified and interviewed ten experienced users and officials from five federal projects that used Agile methods and analyzed and categorized their responses.
section_title: What GAO Found
paragraphs: GAO identified 32 practices and approaches as effective for applying Agile software development methods to IT projects. The practices generally align with five key software development project management activities: strategic planning, organizational commitment and collaboration, preparation, execution, and evaluation. Officials who have used Agile methods on federal projects generally agreed that these practices are effective. Specifically, each practice was used and found effective by officials from at least one agency, and ten practices were used and found effective by officials from all five agencies. The ten practices are:
Start with Agile guidance and an Agile adoption strategy.
Enhance migration to Agile concepts using Agile terms, such as user stories (used to convey requirements), and Agile examples, such as demonstrating how to write a user story.
Continuously improve Agile adoption at both the project level and organization level.
Seek to identify and address impediments at the organization and project levels.
Obtain stakeholder/customer feedback frequently.
Empower small, cross-functional teams.
Include requirements related to security and progress monitoring in your queue of unfinished work (the backlog).
Gain trust by demonstrating value at the end of each iteration.
Track progress using tools and metrics.
Track progress daily and visibly.
GAO identified 14 challenges with adapting and applying Agile in the federal environment:
Teams had difficulty collaborating closely.
Procurement practices may not support Agile projects.
Teams had difficulty transitioning to self-directed work.
Customers did not trust iterative solutions.
Staff had difficulty committing to more timely and frequent input.
Teams had difficulty managing iterative requirements.
Agencies had trouble committing staff.
Compliance reviews were difficult to execute within an iteration time frame.
Timely adoption of new tools was difficult.
Federal reporting practices do not align with Agile.
Technical environments were difficult to establish and maintain.
Traditional artifact reviews do not align with Agile.
Agile guidance was not clear.
Traditional status tracking does not align with Agile.
Finally, officials described efforts to address challenges by clarifying previously unclear guidance on using Agile. In a related effort, the Federal Chief Information Officers (CIO) Council is developing guidance on modular development in the federal government, but it does not specifically address effective practices for Agile.
section_title: What GAO Recommends
paragraphs: GAO is recommending that the Federal CIO Council, working with its chair, OMBs Deputy Director for Management, include practices such as those discussed in this report in the Councils ongoing effort to promote modular development. After reviewing a draft of this report, OMB commented that the recommendation was better addressed to the Council than to its chair. GAO revised the recommendation to address it to the Council working with its chair.

section_title: Letter
paragraphs: Information systems are integral to many aspects of federal government  operations. To support agency missions, the federal government spent at  least $76 billion in fiscal year 2011 on information technology (IT).  However, as we have previously reported, prior IT expenditures too often  have produced disappointing results, including multimillion dollar cost  overruns and schedule delays measured in years, with questionable  mission-related achievements. Congress has expressed long-standing  interest in monitoring and improving federal IT investments, which have  often been developed in long, sequential phases. Recently, several  agencies have tried an alternate approach known as Agile, which calls for  producing software in small, short increments.
Shorter, more incremental approaches to IT development have been  identified as having the potential to improve the way in which the federal  government develops and implements IT. For example, the Office of  Management and Budget (OMB) recently issued guidance that advocates  the use of shorter delivery time frames, an approach consistent with  Agile. experienced users effectively follow this approach and what challenges it  presents in the federal environment.
OMB, 25 Point Implementation Plan to Reform Federal Information Technology  Management, (Washington, D.C.: Dec. 9, 2010) and Immediate Review of Financial  Systems IT Projects, M-10-26 (Washington, D.C.: June 28, 2010).
Accordingly, the objectives of our review were to identify (1) effective  practices in applying Agile for software development solutions and (2)  federal challenges in implementing Agile development techniques.
To identify effective practices in applying Agile for software development  solutions, we interviewed a nongeneralizable sample of experienced Agile  users (see app. III). We identified those users from publications, forums,  and recommendations from federal and private officials knowledgeable  about Agile. To ensure a broad range of experiences, we chose  individuals from private, public, and non-profit backgrounds. We asked  them individually to describe what they have found to be effective  practices in applying Agile methods. We compiled the practices and  asked the users to rate them for effectiveness. We then asked officials  from a nongeneralizable sample of five federal software development  projects that had used Agile methods for their views on the effectiveness  of the practices. The projects were selected to reflect a range of  agencies, system descriptions, and cost. The five federal agencies  supporting these projects were the Departments of Commerce, Defense,  and Veterans Affairs, the Internal Revenue Service, and the National  Aeronautics and Space Administration (see app. IV for additional  information on the projects and responsible officials).
To identify federal challenges in implementing Agile development  techniques, we asked the officials from the five projects to identify  challenges applying Agile in their agency and efforts they had taken to  address these challenges. We analyzed their responses and categorized  them by topic.
We conducted our work from October 2011 through July 2012 in  accordance with all sections of GAO’s Quality Assurance Framework that  were relevant to our objectives. The framework requires that we plan and  perform the engagement to obtain sufficient and appropriate evidence to  meet our stated objectives and to discuss any limitations in our work. We  believe that the information obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives. Further details of  our objectives, scope, and methodology are in appendix I.
subsections: section_title: Background
paragraphs: While federal IT investments can improve operational performance and  increase public interaction with government, too often they have become  risky, costly, and unproductive mistakes. Congress has expressed  interest in monitoring and improving IT investments through hearings and  other reviews over the past two decades. In response, we have testified  and reported on lengthy federal IT projects that too frequently incur cost  overruns and schedule slippages while contributing little to mission- related outcomes. Similarly, in 2010, OMB expressed concern about  expansive federal IT projects that have taken years and have failed at  alarming rates. OMB also noted that many projects follow “grand designs”  to deliver functionality in years, rather than breaking projects into more  manageable chunks and delivering functionality every few quarters.
We recently reported on OMB’s progress on these reforms in GAO, Information  Technology Reform: Progress Made; More Needs to Be Done to Complete Actions and  Measure Results, GAO-12-461 (Washington, D.C.: Apr. 26, 2012). the principal interagency forum for improving agency practices related to  the design, acquisition, development, modernization, use, sharing, and  performance of federal information resources.
Agile software development supports the practice of shorter software  delivery. Specifically, Agile calls for the delivery of software in small, short  increments rather than in the typically long, sequential phases of a  traditional waterfall approach. More a philosophy than a methodology,  Agile emphasizes this early and continuous software delivery, as well as  using collaborative teams, and measuring progress with working  software. The Agile approach was first articulated in a 2001 document  called the Agile Manifesto, which is still used today. The manifesto has  four values: (1) individuals and interactions over processes and tools, (2)  working software over comprehensive documentation, (3) customer  collaboration over contract negotiation, and (4) responding to change  over following a plan. Appendix II provides additional information on the  Agile Manifesto and its related principles.
subsections: section_title: Agile and Waterfall Approaches Differ
paragraphs: The Agile approach differs in several ways from traditional waterfall  software development,of a sequence of phases. For example, the two approaches differ in (1)  the timing and scope of software development and delivery, (2) the timing  and scope of project planning, (3) project status evaluation, and (4)  collaboration.   which produces a full software product at the end
Timing and scope of software development and delivery. In an  Agile project, working software is produced in iterations of typically  one to eight weeks in duration, each of which provides a segment of  functionality. To allow completion within the short time frame, each  iteration is relatively small in scope. For example, an iteration could  encompass a single function within a multistep process for  documenting and reporting insurance claims, such as a data entry  screen or a link to a database. Iterations combine into releases, with  the number of iterations dependent on the scope of the multistep  process. To meet the goal of delivering working software, teams  perform each of the steps of traditional software development for each  iteration. Specifically, for each iteration, the teams identify  requirements, design, and develop software to meet those  requirements, and test the resulting software to determine if it meets  the stated requirements. In contrast, waterfall development proceeds  in sequential phases of no consistent, fixed duration to produce a  complete system, such as one that addresses a comprehensive set of  steps to manage insurance claims. Such full system development  efforts can take several years. Waterfall phases typically address a  single step in the development cycle. For example, in one phase,  customer requirements for the complete product are documented,  reviewed, and handed to technical staff. One or more phases follow,  in which the technical staff develop software to meet those  requirements. In the final phase, the software is tested and reviewed  for compliance with the identified requirements.
Timing and scope of project planning. In Agile, initial planning  regarding cost, scope, and timing is conducted at a high level.  However, these initial plans are supplemented by more specific plans  for each iteration and the overall plans can be revised to reflect  experience from completed iterations. For example, desired project  outcomes might initially be captured in a broad vision statement that  provides the basis for developing specific outcomes for an iteration.  Once an iteration has been completed, the overall plans can be  revised to reflect the completed work and any knowledge gained  during the iteration. For example, initial cost and schedule estimates  can be revised to reflect the actual cost and timing of the completed  work. In contrast, in traditional waterfall project management, this  analysis is documented in detail at the beginning of the project for the  entire scope of work. For example, significant effort may be devoted  to documenting strategies, project plans, cost and schedule  estimates, and requirements for a full system.
Project status evaluation. In Agile, project status is primarily  evaluated based on software demonstrations. For example, iterations  typically end with a demonstration for customers and stakeholders of  the working software produced during that iteration. The  demonstration can reveal requirements that were not fully addressed  during the iteration or the discovery of new requirements. These  incomplete or newly-identified requirements are queued for possible  inclusion in later iterations. In contrast, in traditional project  management, progress is assessed based on a review of data and  documents at predetermined milestones and checkpoints. Milestones  and checkpoints can occur at the end of a phase, such as the end of  requirements definition, or at scheduled intervals, such as monthly.  The reviews typically include status reports on work done to date and  a comparison of the project’s actual cost and schedule to baseline  projections. Federal IT evaluation guidance, such as our IT  Investment Management guidance and OMB IT reporting  requirements specify evaluations at key milestones, and annually,  which more closely align with traditional development methods. For  example, for major projects, OMB requires a monthly comparison of  actual and planned cost and schedule and risk status and annual  performance measures using, for example, earned value  management (EVM).
Collaboration. Agile development emphasizes collaboration more  than traditional approaches do. For example, to coordinate the many  disciplines of an iteration, such as design and testing, customers work  frequently and closely with technical staff. Furthermore, teams are  often self-directed, meaning tasks and due dates are done within the  team and coordinated with project sponsors and stakeholders as  needed to complete the tasks. In contrast, with traditional project  management, customer and technical staff typically work separately,  and project tasks are prescribed and monitored by a project manager,  who reports to entities such as a program management office.
See figure 1 for a depiction of Agile development compared to waterfall  development.
There are numerous frameworks available to Agile practitioners. One  framework, called eXtreme Programming (XP), includes development  techniques. Another framework, called Scrum, defines management  processes and roles. The Scrum framework is widely used in the public  and private sector, and its terminology is often used in Agile discussions.  For example, Scrum iterations are called sprints, which are bundled into  releases. Sprint teams collaborate with minimal management direction,  often co-located in work rooms. They meet daily and post their task status  visibly, such as on wall charts.
Other concepts commonly used by sprint teams are user stories, story  points, and backlog. User stories convey the customers’ requirements. A  user story typically follows the construct of “As a <type of user> I want  <some goal> so that <some reason>.” For example, “As a claims  processor, I want to check a claim payment status so that I can promptly  reply to a customer’s request for payment status.” Each user story is  assigned a level of effort, called story points, which are a relative unit of  measure used to communicate complexity and progress between the  business and development sides of the project. To ensure that the  product is usable at the end of every iteration, teams adhere to an  agreed-upon definition of done. This includes stakeholders defining how  completed work conforms to an organization’s standards, conventions,  and guidelines. The backlog is a list of user stories to be addressed by  working software. If new requirements or defects are discovered, these  can be stored in the backlog to be addressed in future iterations.
Progress in automating user stories is tracked daily using metrics and  tools. An example of a metric is velocity. Velocity tracks the rate of work  using the number of story points completed or expected to be completed  in an iteration. For example, if a team completed 100 story points during a  four-week iteration, the velocity for the team would be 100 story points  every four weeks. An example of a tool is a burn-down chart, which tracks  progress and the amount of work remaining for an iteration or for a  release, which is made up of multiple iterations.
section_title: Agile in the Private and Federal Sectors
paragraphs: Agile use is reported in the private sector for small to medium sized  projects and is starting to be used for larger projects as well. Also, widely  accepted industry guidance on software development has recently been  revised to include more Agile approaches. Specifically, the Software  Engineering Institute’s Capability Maturity Model® Integration updated  some process areas to help those using Agile to interpret its practices.
Furthermore, the federal government has begun to use Agile. For  example, we have reported on several federal software development  efforts that have used Agile techniques. Specifically, in December 2010  we reported that the Department of Veterans Affairs was using Agile to  develop software to support a new benefit for veterans. We also  reported that the Department of Defense was developing the Global  Combat Support System-Joint system using Agile. In addition, the  department sponsored studies that examined the possibility of more  widespread use of Agile in its development projects.
section_title: Effective Practices for Applying Agile
paragraphs: We identified 32 practices and approaches as effective for applying  Agile to software development projects, based on an analysis of practices  identified by experienced Agile users. Our analysis also found that the  identified practices generally align with five key project management  activities outlined in widely-accepted software development guidance:  strategic planning, organizational commitment and collaboration,  preparation, execution, and evaluation.
subsections: section_title: Strategic Planning
paragraphs: Strategic planning describes an organization’s overall plans in an Agile  environment. Six practices align with strategic planning. They are:
Strive to be more Agile, rather than simply following Agile  methods and steps. This approach encourages adoption of the  philosophy, or mindset, rather than specific steps. This is also referred  to as being Agile, or having agility versus using it.
Allow for a gradual migration to Agile appropriate to your  readiness. Migration steps might include combining Agile and  existing methods, conducting pilots, and preparing technical  infrastructure.
Observe and communicate with other organizations  implementing Agile. For example, those starting to use Agile can  consult with others who have more experience, including academic,  private sector, and federal practitioners.
Follow organizational change disciplines, such as establishing a  sense of urgency and developing a change vision. A clear vision  of change helps staff understand what the organization is trying to  achieve. Another organizational change discipline is communication  strategies.
Be prepared for difficulties, regression, and negative attitudes.
This approach reinforces that Agile is not painless and users may  backslide to entrenched software methods.
Start with Agile guidance and an Agile adoption strategy. This  practice advocates having these elements in place at the start, even if  they must be copied from external sources.
section_title: Organizational Commitment and Collaboration
paragraphs: Organizational commitment describes the management actions that are  necessary to ensure that a process is established and will endure.  Collaboration in Agile typically refers to the close and frequent interaction  of teams. Four practices align with organizational commitment and  collaboration:
Ensure all components involved in Agile projects are committed  to the organization’s Agile approach. This practice encourages  organizations to ensure that everyone contributing to a project  understands and commits to the organization’s approach. This  includes those working directly on the project and those with less  direct involvement, such as those providing oversight.
Identify an Agile champion within senior management. This  practice calls for someone with formal authority within the organization  to advocate the approach and resolve impediments at this level.
Ensure all teams include coaches or staff with Agile experience.
This practice stresses the importance of including on each team those  with direct experience in applying Agile. While training is helpful,  hands on experience helps the team members learn and adjust.
Empower small, cross-functional teams. Empowered teams of 7 to  18 people decide what to deliver and how to produce it. The teams  should not over-rely on one member’s skills.
section_title: Preparation
paragraphs: Taking certain preparatory steps prior to the start of an iteration can  facilitate a rapid development pace. The following eight practices  generally align with the preparation of people and processes.
Train the entire organization in your Agile approach and mindset,  and train Agile practitioners in your Agile methods. For example,  managers must understand the approach so that they know how it will  affect them and teams need to know the specific steps of an iteration  to conduct it properly.
Ensure that subject matter experts and business team members  have the required knowledge. This practice stresses that staff  involved in fast-paced iterations must truly be experts in the  processes being automated in that iteration in order to reduce delays.  For example, a team member representing financial customers must  be fully familiar with the needs of those customers.
Enhance migration to Agile concepts using Agile terms and  examples. For example, use terms like user stories instead of  requirements, and Agile Center of Excellence instead of Project  Management Office. Provide examples, such as one illustrating the  small scope of a user story to teams writing these stories.
Create a physical environment conducive to collaboration. A  common practice is to co-locate the team in a single room where they  can continually interact. Other ways to enhance collaboration are to  reorganize office space and use tools to connect remote staff.
Identify measurable outcomes, not outputs, of what you want to  achieve using Agile. An example of this practice is creating a vision  statement of project outcomes (such as a decrease in processing time  by a specific percent in a set time), rather than outputs (such as the  amount of code produced).
Negotiate to adjust oversight requirements to a more Agile  approach. This practice notes that teams may be able to adjust  oversight requirements by using frequent, tangible demonstrations to  gain the trust of reviewers and investors, potentially reducing the need  for more formal oversight documents.
Ensure that the definition of how a story will be determined to be  done is comprehensive and objective. Comprehensiveness  includes defining what constitutes a finished product (i.e., packaged,  documented, tested, and independently verified). Objective means  measurable or verifiable versus subjective judgment.
Make contracts flexible to accommodate your Agile approach.  Contracts requiring waterfall-based artifacts and milestone reviews  may not support the frequent changes and product demonstrations in  iterations, and may inhibit adoption.
section_title: Execution
paragraphs: Execution entails the concrete steps necessary to conduct the iteration  following the designated approach. The seven identified practices that  align with execution are:
Use the same duration for each iteration. An example would be  establishing that iterations will be four weeks each within a release to  establish a uniform pace.
Combine Agile frameworks such as Scrum and XP if appropriate.  Disciplines from different frameworks can be combined. For example,  use project management disciplines from Scrum and technical  practices from XP.
Enhance early customer involvement and design using test- driven development. Test-driven development refers to writing  software code to pass a test. This practice maintains that involving  customers in these tests helps to engage them in the software  development process.
Include requirements related to security and progress  monitoring in your queue of unfinished work (backlog). Including  activities such as security reviews and status briefings in the backlog  ensures their time and cost are reflected and that they are addressed  concurrent with, and not after, iteration delivery.
Capture iteration defects in a tool such as a backlog. This practice  calls for queuing issues so that they are resolved in later iterations.  For example, lists of unmet requirements generated at end-of-iteration  demonstrations should be queued in the backlog for correction in a  future iteration.
Expedite delivery using automated tools. For example, tools can  track software modifications, and compliant development sites or  “sandboxes” help customers conceptualize the software in an  environment that meets architectural and security standards.
Test early and often throughout the life cycle. The theme of this  practice is that testing during software code delivery instead of after  delivery reduces risk and remediation costs.
section_title: Evaluation
paragraphs: Evaluations can occur at the project and organizational level. For  example, at the project level, the iteration is reviewed at its completion in  a retrospective. At the organizational level, processes are reviewed for  opportunities to improve the approach. The following seven practices  align with evaluation:
Obtain stakeholder/customer feedback frequently and closely.
For example, feedback is obtained during the iteration and at its  completion at an iteration retrospective. This practice was linked to  reducing risk, improving customer commitment, and improving  technical staff motivation.
Continuously improve Agile adoption at both the project level  and organization level. This practice invokes the discipline of  continuous improvement, meaning always looking for ways to  improve. For example, improvements can be made by adding  automated test and version control tools, and enhancing team rooms.  These issues can be tracked in project and organizational-level  backlogs.
Seek to identify and address impediments at the organization  and project levels. This practice encourages organizations to be  frank about identifying impediments so that they can be addressed.
Determine project value based on customer perception and  return on investment. This practice recognizes that tracking  progress only against cost or schedule criteria set before the project  began could lead to inaccurate measurement of progress if, for  example, major changes in scope occur. Instead, Agile encourages  customer feedback as one measure of progress. Comparing solution  value to the cost of the solution is also a gauge of success.
Gain trust by demonstrating value at the end of each iteration.
This practice includes demonstrating key requirements in early  iterations, and showing customers that requirements in the backlog  are delivered and not forgotten.
Track progress using tools and metrics. Progress can be tracked  using tools and metrics such as burn-down charts and velocity, which  can be automated, and by success indicators such as “customer  delight,” and reduced staff stress and overtime.
Track progress daily and visibly. This practice stresses that status  is checked daily and publicly. For example, a progress chart is posted  openly in the team’s workspace, with timely revisions to reflect  ongoing feedback.
section_title: Federal Use of Effective Practices
paragraphs: Officials who have used Agile on federal projects at five agencies  generally agreed that the practices identified by the experienced users  are effective in a federal setting. Specifically, each practice was used and  found effective by officials from at least one agency. Ten of the 32  practices were used and found effective by officials at all five agencies  (see table 1).
Also, in most cases, a practice was still believed to be effective even if it  was not used. For example, officials explained that they did not use a  practice they indicated was effective because it was not appropriate for  their project or that they used an alternate practice.
Although the identified practices were generally described as effective,  officials from three agencies each reported one practice they had used  but found to be not effective. According to the agency officials, two  practices were identified as ineffective because they were difficult to  implement. These practices were: (1) ensuring commitment from  components and (2) negotiating oversight requirements. The third  practice, striving to be Agile rather than simply following Agile methods,  was described by an agency official as not effective because he believed  that strict adherence was necessary for a successful project.
section_title: Federal Challenges in Applying Agile
paragraphs: We identified 14 challenges with adapting to and applying Agile in the  federal environment based on an analysis of experiences collected from  five federal agencies that had applied Agile to a development effort.  These challenges relate to significant differences in not only how software  is developed but also how projects are managed in an Agile development  environment versus a waterfall development environment. We aligned the  challenges with four of the project management activities used to  organize effective practices: (1) ensuring organizational commitment and  collaboration, (2) preparing for Agile, (3) executing development in an  Agile environment, and (4) evaluating the product and project. In addition  to identifying challenges, federal officials described efforts underway at  their agencies to address these challenges.
subsections: section_title: Organizational Commitment and Collaboration
paragraphs: As described in the effective practices, Agile projects require the ongoing  collaboration and commitment of a wide array of stakeholders, including  business owners, developers, and security specialists. One way Agile  promotes commitment and collaboration is by having teams work closely  together, in one location, with constant team communication. Officials at  the selected agencies identified challenges in achieving and maintaining  such commitment and collaboration from their stakeholders as follows.
Teams had difficulty collaborating closely: Officials from three  agencies reported that teams were challenged in collaborating  because staff were used to working independently. For example, one  official reported that staff were challenged when asked to relocate to a  team room because the technical staff preferred to work alone. The  official added that some staff viewed open communication, such as  posting project status on team room wall charts, as intrusive. A  second official said that technical staff did not like constantly showing  their work to customers. The third official said that customers initially  did not want to see such development, preferring to wait for a polished  product.
Teams had difficulty transitioning to self-directed work: Officials  at two agencies reported that staff had challenges in transitioning to  self-directed teams. In Agile, teams made up of customers and  technical staff are encouraged to create and manage their tasks  without project manager direction and to elevate issues to  stakeholders who have the authority to resolve them. Cross  functionality is also encouraged to allow teams to share tasks. One  official reported that teams used to direction from a project manager  were challenged in taking responsibility for their work and in elevating  issues they could not resolve within the team to senior officials. A  second official noted that it was a challenge to create cross-functional  teams because federal staff tend to be specialists in one functional  area. An example of this would be where a team could include  someone to represent system users, but that person may not be  familiar with the needs of all users. Specifically, a team developing an  insurance system might include someone with a background in claims  processing. However, that person may not be experienced with  payment procedures.
Staff had difficulty committing to more timely and frequent input:  While Agile advocates frequent input and feedback from all  stakeholders, four agency officials noted challenges to commit to  meeting such input expectations. One agency official noted that  individuals were challenged to commit to keeping work products, such  as schedules, updated to reflect the status of every iteration because  they were not used to this rapid pace. A second official stated that  teams initially had difficulty maintaining the pace of an iteration  because they were used to stopping their work to address issues  rather than making a decision and moving on. A third official said that  it was challenging incorporating security requirements at the rapid  pace of the sprint. A fourth official said customer availability was a  challenge because customers initially did not understand the amount  and pace of the time commitment for Agile and needed to develop a  mindset to attend meetings as well as frequently review deliverables.
Agencies had trouble committing staff: Three agency officials  reported being challenged assigning and maintaining staff  commitments to projects. The frequent input expected of staff involved  in projects requires a more significant time commitment than that  required for waterfall development projects that allow more sporadic  participation. For example, two officials said their agencies were  challenged dedicating staff with multiple, concurrent duties to teams  because staff could not be spared from their other duties while  participating in the Agile teams. The third official said stakeholder  commitment is challenging to maintain when stakeholders rotate  frequently and new staff need to learn the roles and responsibilities of  those being replaced.
section_title: Preparation
paragraphs: When an organization following waterfall software development migrates  to Agile, new tools and technical environments may be required to  support that approach, as well as updates to guidance and procurement  strategies. Officials described challenges in preparing for Agile as follows.
Timely adoption of new tools was difficult: As identified in the  effective practices, automated tools may be used to support project  planning and reporting. One official noted that implementing Agile  tools that aid in planning and reporting progress was initially a  challenge because there was a delay in buying, installing, and  learning to use these tools.
Technical environments were difficult to establish and maintain:  Two agency officials noted that establishing and maintaining technical  environments posed challenges because Agile calls for development,  test, and operational activities to be performed concurrently.  According to one agency’s officials, preparing and maintaining  synchronized hardware and software environments for these three  activities in time to support the releases was expensive to support and  logistically challenging. Furthermore, one of these officials noted that  his agency experienced a challenge running multiple concurrent  iterations because this required more complex coordination of staff  and resources.
Agile guidance was not clear: Officials from three agencies  identified a challenge related to the lack of clear guidance for Agile  software development, particularly when agency software  development guidance reflected a waterfall approach. For example,  one official said that it was challenging to develop policy and  procedure guidance for iterative projects because they were new, and  the agency strategy aligned with the waterfall approach. As a result, it  was difficult to ensure that iterative projects could follow a standard  approach. A second official reported that deviating from waterfall- based procedural guidance to follow Agile methods made people  nervous. For example, staff were nervous following team versus  project manager directed tasks because this approach was not in their  IT guidance. A third official said that their guidance mixed iterative and  waterfall life cycle approaches, which staff found confusing.
Procurement practices may not support Agile projects: Agile  projects call for flexibility adding the staff and resources needed to  meet each iteration, and to adapt to changes from one iteration to the  next. One official stated that working with federal procurement  practices presents a challenge where they do not support the flexibility  required. For example, he said that federal contracts that require  onerous, waterfall-based artifacts to constantly evaluate contractor  performance are not needed in an Agile approach when the contractor  is part of the team whose performance is based on the delivery of an  iteration. Furthermore, the official said that they are challenged  changing contractor staff in time to meet iteration time frames and that  accommodating task changes from one iteration to the next is  challenging because contracting officers require cumbersome  traditional structured tasks and performance checks.
section_title: Execution
paragraphs: As described in the effective practices, Agile projects develop software  iteratively, incorporating requirements and product development within an  iteration. Such requirements may include compliance with agency legal  and policy requirements. Officials reported challenges executing steps  related to iterative development and compliance reviews as follows.
Customers did not trust iterative solutions: Agile software  products are presented to customers incrementally, for approval at the  end of each iteration, instead of presenting complete products for  approval at waterfall milestones. Officials at two agencies reported a  challenge related to customer mistrust of iterative solutions.  Specifically, one agency official said customers expecting a total  solution feared that the initial demonstrations of functionality provided  in the current iteration would be considered good enough, and they  would not receive further software deliveries implementing the  remainder of their requirements. At another agency, an official said  this fear contributed to customers finding it difficult to define done.  Specifically, customers were challenged in defining when each  requirement would be considered done because they were afraid that  this would be viewed as meaning all related functions were being met,  and that unmet requirements would be dropped and never  implemented.
Teams had difficulty managing iterative requirements: Teams  provide input on prioritizing requirements, and deciding what to do  with new requirements discovered during iterations. Two agencies’  officials reported challenges managing requirements. Specifically, one  official reported that customers were initially challenged to validate  and prioritize which requirements would be assigned to a release.  Using the waterfall development model, they were used to identifying  all requirements up front and not revisiting them as they were  developed. The second official said they were challenged to  accommodate new requirements within the fixed schedule for a  product release.
Compliance reviews were difficult to execute within an iteration  time frame: Iterations may incorporate compliance reviews to ensure,  for example, that agency legal and policy requirements are being met  within the iteration. One agency official reported a challenge obtaining  compliance reviews within the short, fixed time frame of an iteration  because reviewers followed a slower waterfall schedule. Specifically,  the official said that compliance reviewers queued requests as they  arose and that the reviews could take months to perform. This caused  delays for iterations that needed such reviews within the few weeks of  the iteration.
section_title: Evaluation
paragraphs: Agile advocates evaluation of working software over the documentation  and milestone reporting typical in traditional project management.  Officials described challenges in evaluating projects related to the lack of  alignment between Agile and traditional evaluation practices. Specifically,  officials explained that:
Federal reporting practices do not align with Agile: Two agency  officials noted that several federal reporting practices do not align with  Agile, creating challenges. For example, one official said federal  oversight bodies want status reports at waterfall-based milestones  rather than timely statements regarding the current state of the  project. The second official said OMB’s IT investment business case  (known as the exhibit 300) and IT Dashboard, a publicly available  website that displays detailed information on federal agencies’ major  IT investments, are waterfall-based. For example, the IT Dashboard  calls for monthly statistics instead of demonstrations of working  software. He also noted that it is frustrating when dashboard statistics  are flagged in red to note deviations, even when the deviation is  positive, such as being ahead of schedule and under cost.
Traditional artifact reviews do not align with Agile: Traditional  oversight requires detailed artifacts in the beginning of a project, such  as cost estimates and strategic plans, while Agile advocates  incremental analysis. One agency official noted that requiring these  artifacts so early was challenging because it was more worthwhile to  start with a high-level cost estimate and vision to be updated as the  solution was refined through iterations, rather than spending time  estimating costs and strategies that may change.
Traditional status tracking does not align with Agile: Officials from  three agencies noted that project status tracking in Agile does not  align with traditional status tracking methods, creating challenges. For  example, one official said that tracking the level of effort using story  points instead of the traditional estimating technique based on hours  was a challenge because team members were not used to that  estimation method, although eventually this method was embraced.  Two other agency officials said EVM was challenging to apply in an  Agile environment. Specifically, one official said that the required use  of EVM was challenging because there was no guidance on how to  adapt it to iterations. The second official found EVM challenging  because the agency was required to use it to track changes in cost,  schedule, and product scope through monthly reports, and changes  were viewed as control problems rather than as revisions to be  expected during an iteration. For example, the project’s scope was  prioritized within every iteration based on the cost and schedule limits  of the iteration and release. He also noted that risk tracking in Agile  does not align with traditional risk tracking methods because issues  are addressed within an iteration rather than queued, such as in a  traditional monthly risk log.
In addition to identifying challenges, federal officials described their efforts  to address these challenges. For example, officials said they clarify  policies to address the challenge of Agile guidance lacking clarity. To  mitigate the challenge related to customers not trusting iterative solutions,  an official said they call the iteration review a mini-critical design review.  This helps customers understand that they must declare the iteration  complete or not, known as committing to done. Another official said one  way that they addressed the challenge related to teams having difficulty  managing iterative requirements was to add an empty iteration to the end  of the release schedule to accommodate requirements discovered during  the iterations.
In addition to the efforts at individual agencies to mitigate Agile  challenges, the Federal CIO Council has begun an effort on a related  topic. According to an official working with the Council, it is currently  drafting a document on modular development. Consistent with OMB’s IT  reform efforts, the document is expected to provide guidance for agencies  seeking to use more modular development approaches, such as Agile.  However, according to the official, the draft does not specifically address  Agile effective practices. Also, in June 2012 OMB released contracting  guidance to support modular development. This guidance includes  factors for contracting officers to consider for modular development efforts  regarding for example, statements of work, pricing arrangements, and  small business opportunities.
section_title: Conclusions
paragraphs: As Agile methods begin to be more broadly used in federal development  projects, agencies in the initial stages of adopting Agile can benefit from  the knowledge of those with more experience. The ongoing effort by the  Federal CIO Council to develop guidance on modular development  provides an excellent opportunity to share these experiences. The  effective practices and approaches identified in this report, as well as  input from others with broad Agile experience, can inform this effort.
section_title: Recommendation for Executive Action
paragraphs: To ensure that the experiences of those who have used Agile  development are shared broadly, we recommend that the Federal CIO  Council, working with its chair, the Office of Management and Budget’s  Deputy Director for Management, include practices such as those  discussed in this report in the Council’s ongoing effort to promote modular  development in the federal government.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of our report to OMB and to the five federal agencies  included in our review. In oral comments on the draft, OMB’s E- government program manager said that the draft recommendation was  better addressed to the Federal CIO Council than to the OMB official who  is the chair of the Council. Accordingly, we revised the recommendation  to address it to the Council, working with its chair, the OMB Deputy  Director for Management. Two of the five agencies provided written  comments on the draft, which are reprinted in appendix V and VI.  Specifically, the Department of Veterans Affairs Chief of Staff stated that  the department generally agreed with the draft’s findings, and the Acting  Secretary of the Department of Commerce stated that the Patent and  Trademark Office concurred with our assessment. Two other agencies,  the Internal Revenue Service and the Department of Defense, provided  technical comments via e-mail, which we incorporated as appropriate. In  an e-mail, a manager in the National Aeronautics and Space  Administration (NASA) center included in our review said that NASA had  no comments.
As agreed with your offices, we will send copies of this report to  interested congressional committees; the Secretaries of Defense,  Commerce, and Veterans Affairs; the Administrator of NASA and the  Commissioner of Internal Revenue; the Director of the Office of  Management and Budget; and other interested parties. In addition, the  report will be available at no charge on our website at http://www.gao.gov.
If you or your staff have any questions on the matters discussed in this  report, please contact David A. Powner at (202) 512-9286 or Dr.  Nabajyoti Barkakati at (202) 512-4499 or by e-mail at pownerd@gao.gov  or barkakatin@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made major contributions to this report are listed in  appendix VII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to identify (1) effective practices in applying Agile for  software development solutions and (2) federal challenges in  implementing Agile development techniques.
To identify effective practices, we interviewed a nongeneralizable sample  of nine experienced users and a tenth experienced user helped us pre- test our data collection process. To identify these users, we researched  publications, attended forums, and obtained recommendations from  federal and private officials knowledgeable about Agile. We selected  individuals with Agile software development experience with public,  private sector, and non-profit organizations. Using a structured interview,  we asked them to identify effective practices when applying Agile  methods to software development projects. We then compiled the  reported practices and aligned and combined some with a broader  practice. For example, practices related to preparation, such as mock and  pilot iterations, were aligned and then combined into the final practice,  “Allow for a gradual migration to Agile appropriate to your readiness.” If a  practice did not align with other or broader practices, it was listed  individually.
We then sent the resulting list of practices in a questionnaire to our  experienced users. This list was not organized into categories to ensure  that each practice would be viewed individually. We asked our users to  rate each practice as either (1) highly effective, (2) moderately effective,  (3) somewhat effective, or (4) not applicable/do not know. We compiled  the ratings and included in our list the practices that received at least six  ratings of highly effective or moderately effective from the 8 experienced   This resulted in 32 practices,  users who provided the requested ratings. which we aligned to key project management activities in Software  Engineering Institute guidance: strategic planning, organizational  commitment and collaboration, preparation, execution, and evaluation.  This alignment was based on our best judgment.
The ninth experienced user was asked for input on the list of practices with the others,  but did not respond in time to meet our reporting deadline.
To identify federal challenges, we interviewed officials responsible for five  federal software development projects that reported using Agile practices.  To identify the projects, we researched our previous work, federal  websites, and publications, and attended federal forums. We selected a  nongeneralizable sample of projects designed to reflect a range of  agencies, system descriptions, and cost (see app. IV for details about the  projects and the responsible officials). We then asked officials from each  project to identify federal challenges in implementing an Agile approach  using a structured interview. We summarized the challenges and  categorized them as aligning with either organizational commitment and  collaboration, preparation, execution, or evaluation. Separately, we sent  the federal officials a questionnaire listing the effective practices we  compiled based on input from our experienced users. The questionnaire  asked whether these practices were used and found effective. Although  our results are not generalizable to the population of software  development projects reporting the use of Agile practices, they provided  valuable insight into both the effective use and challenges in applying  Agile in the federal sector.
We conducted our work from October 2011 through July 2012 in  accordance with all sections of GAO’s Quality Assurance Framework that  are relevant to our objectives. The framework requires that we plan and  perform the engagement to obtain sufficient and appropriate evidence to  meet our stated objectives and to discuss any limitations in our work. We  believe that the information obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: The Agile Manifesto and Principles
paragraphs: Agile development encompasses concepts that were previously used in  software development. These concepts were documented as Agile  themes and principles by 17 practitioners, who called themselves the  Agile Alliance. In February 2001 the Alliance released “The Agile  Manifesto,” in which they declared: “We are uncovering better ways of  developing software by doing it and helping others do it. Through this  work we have come to value:  individuals and interactions over processes and tools working software over comprehensive documentation  customer collaboration over contract negotiation  responding to change over following a plan.”
The Alliance added that while they recognized the value in the second  part of each statement (i.e., “processes and tools”), they saw more value  in the first part (“individuals and interactions”). The Alliance further  delineated their vision with twelve principles.
Our highest priority is to satisfy the customer through early and  continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile  processes harness change for the customer’s competitive advantage.
Deliver working software frequently, from a couple of weeks to a  couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout  the project.
Build projects around motivated individuals. Give them the  environment and support they need, and trust them to get the job  done.
The most efficient and effective method of conveying information to  and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors,  developers, and users should be able to maintain a constant pace  indefinitely.
Continuous attention to technical excellence and good design  enhances agility.
Simplicity—the art of maximizing the amount of work not done—is  essential.
The best architectures, requirements, and designs emerge from self- organizing teams.
At regular intervals, the team reflects on how to become more  effective, then tunes and adjusts its behavior accordingly.
section_title: Appendix III: Experienced Users
section_title: Appendix IV: Federal Project Profiles
paragraphs: The five federal software development projects that reported challenges  in applying Agile practices are profiled as follows.
section_title: Appendix V: Comments from the Department of Veterans Affairs
section_title: Appendix VI: Comments from the Department of Commerce
section_title: Appendix VII: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contact names above, individuals making contributions  to this report included James R. Sweetman, Jr. (assistant director), Jenny  Chanley, Neil Doherty, Rebecca Eyler, Claudia Fletcher, Nancy Glover,  and Karl Seifert.