The elusive silver lining: how we fail to learn from failure in software development

As modem organizations struggle with increasing complexity and difficult challenges, we argue that every experience — win, lose, or draw -is a valuable organizational asset that must be fully exploited. First, this requires an organization willing to view failures as opportunities to learn something rather than as embarrassing moments to be quickly forgotten. Second, and often missing, organizations need a formal postmortem diagnostic tool that can reliably discern what worked on a project and what did not. Without such a tool, it is possible that the wrong lessons will be learned. We describe our development of and experiences with a system dynamics based tool for postmortem project diagnosis. In this paper we focus our attention on software development projects since they have become critical to the successful operation and strategy of many organizations, yet have experienced serious problems. As an example, when applied to a NASA software development project, three types of insights were gained. First, we show how intuition alone may not be sufficient to handle the complex and dynamic interactions characterizing the software project environment, and may indeed mislead us in deriving the wrong lesson regarding staffing policy. Second, we show how the experimentation capability of the model may be utilized to derive additional learning "dividends" from what continues to be a seriously under-exploited software project lesson regarding undersizing. Third, the model uncovers a well-disguised lesson, a case where too much of a good thing, such as quality assurance, was bad. INTRODUCTION There is a silver lining to every failure. For it is only through costly experience and errors that managers can develop effective intuitive judgement. "Good judgement is usually the result of experience. And experience is frequently the result of bad judgement" (Neustadt and May, 1986). And, not unlike sticking one's hand into the fire, what we leam from our mistakes is often more indeUble. Why do we fail to leam from project failures? There are at least two reasons. First, we rarely try. Generally, mistakes are hidden rather than reported and evaluated. Second, and often missed, the important lessons to be learned are rarely conveniently packaged for easy picking, rather they often need to be dug out from deep within the project experience. A primary objective of this article is to show how and why the silver lining often eludes us. To illustrate this we will focus on the information technology field and use a case study of a real software project Software technology is playing an ever larger role in formulating business strategy, in determining how an organization operates, how it creates its products, and indeed in reshaping the product itself [(Porter and Millar, 1985) and (Bales, 1988)]. Yet, to the dismay of not only system professionals but business executives as well, this formidable dependance on software technology has not been matched by a corresponding maturity in the capability to manage it. We continue to produce too many project failures, marked by cost overruns, late deliveries, poor reliability, and users' dissatisfaction (Newport, 1986). Failure to learn from mistakes has been a major obstacle to improving software project management: We talk about software engineering but reject one of the most basic engineering practices: identifying and learning from our mistakes. Errors made while building one system appear in the next one. What we need to remember is the attention given to failures in the more established branches of engineering (Boddie, 1987). We will demonstrate the utility of a system dynamics based tool for conducting a postmortem diagnostic analysis of what worked on a software project and what did not. THE CHALLENGE OF SOFTWARE DEVELOPMENT One measure of the impact of software is on the pocketbook. It has been estimated that U.S. expenditures for software development and maintenance wUl grow to more than $225 billion by 1995 in the U.S. and more than $450 biUion worldwide (Boehm, 1987). This growth in demand for software has not, however, been painless. The record shows that the software industry has been marked by cost overruns, late deliveries, poor reliability, and users' dissatisfaction, collectively referred to as the "software crisis". As early as November 9, 1979, a report to Congress by the Comptroller General cited the dimensions of the "software crisis" within the federal government. The report's tide summarizes the issue: "Contracting for Computer Software Development— Serious Problems Require Management Attention to Avoid Wasting Additional Millions. " The report concludes, "The government got for its money less than 2 percent of the total value of the contracts. " More than a decade later, the problems persisted . An article in the December 18, 1989 issue oi Defense News described the software problems with the Peace Shield project which was then four years behind schedule and estimated to be up to $300 million over budget (Baker and Silverberg, 1989). Big as the direct costs of the"software crisis" are, the indirect costs can be even bigger, because software is often on the critical path in overall system development. That is, any slippages in the software schedule translate directly into slippages in the overall delivery schedule of the system (e.g. a new approach to customer order entry or manufacturing scheduling). Although many of the largest'and most completely documented examples are found in military projects, the "software crisis" is by no means confined to projects developed by or for the federal government. It is similarly prevalent within private sector organizations (Zmud, 1980). For example, DeMarco noted: • Fifteen percent of all software projects never deliver anything; that is, they fail utterly to achieve their established goals. • Overruns of one hundred to two hundred percent are common in software projects. So many software projects fail in some major way that we have had to redefine "Success" to keep everyone from becoming despondent. Software projects are sometimes considered successful when the overruns are held to thirty percent or when the user only junks a quarter of the result. Software people are often willing to call such efforts successes, but members of our user community are less forgiving. They know failure when they see it (DeMarco, 1987). Personal computer software development is not immune to these problems either. The headline in the May 11, 1990 issue of The Wall Street Journal says it explicitly: "Creating New Software was Agonizing Task for Mitch Kapor Firm" (Carroll, 1990). The article was sub-tided: "Despite Expert's Experience Job Repeatedly Overran Time and Cost Forecasts. " The issues cited in that article (e.g., ". programmers spend 90% of their time on the first 80% of a project and 90% of their time on the final 20%") have been repeated for decades in both public and private organizations, large and small companies (Kapor's company had less than 30 people), and computers firom mainframes to PC's. Due to the embarrassment and bad publicity associated with such problems, it is likely that only a small portion are ever publicly reported. Mitch Kapor, the founder of Lotus Development Corporation, agreed to describe the experiences in his new company ON Technology Inc. "because he believes that software design must be improved and the development process better understood. " (Carroll, 1990) That nicely sums up a goal of this paper. Disclosing the failure is only the first step, learning from the failure is the critical next step. ACASE-STIJDY Consider the case of NASA's DE-A software project. The project was conducted at the Systems Development Section of the Goddard Space Flight Center (GSFC) in Greenbelt, Maryland to design, implement, and test a software system for processing telemetry data and providing attitude determination and control for the DE-A satellite. In planning and managing this project, approximately 85% of the total project cost was allocated to development (design and coding) and the remaining 15% to testing. Furthermore, a relatively large portion (30%) of the development effort was allocated to quality assurance (QA), a level that is significantly higher than the industry norm (Boehm, 1981). Initially, the project was estimated to be 16,000 delivered source instructions (DSI) in size, and its cost and schedule were estimated to be 1,100 man-days and 320 working days, respectively. Figure 1 depicts the values of these and other DE-A project variables over the duration of the project. Because NASA's launch of the DE-A satellite was tied to the completion of the DE-A software, serious schedule slippages could not be tolerated. Specifically, all software was required to be accepted and frozen three months before launch. As the project slipped and this date approached, management reacted by adding new people to the project to meet the strict launch deadline . as evidenced by the rising workforce pattern curve in the final stages of the project seen in Figure 1. The actual final results were: 24,400 DSI, 2,200 man-days, and 380 days. Compared with its original estimates, obviously the DE-A project is not a total success. The project overshot its schedule by 20% and its cost by 100%. On the positive side, the end product was reported to be of high quality i.e., was reliable, stable, and easy to maintain (NASA, 1983). Relying upon "conventional wisdom", what lessons might NASA have learned from the DE-A project? Staffing . Obviously, the DE-A staffing policy of continuing to add people late into the project is not cost effective. Indeed, Brooks (1975) suggests that by adding new people to the late DE-A project, management actually delayed it funher! Thus, in the future they should limit hiring to the early phases only. Undersizing . Another obvious culprit is the initial 35% underestimation of the product's size. Schedule estimatio