Managing Priorities : A Key to Systematic Decision Making

A central concern of systems engineering is selecting a most preferred alternative from among competing choices. The selection process is sometimes called tradeoff analysis, and is built on the methods of decision analysis and utility theory. The process can be loosely divided into two parts, a first part in which one determines the relative priority of various objectives that will be fulfilled, or not, by the alternatives; and a selection phase in which alternatives are compared, and a preferred alternative chosen. This paper discusses the means of determining the priority order for implementing system changes. It is concerned with the priority ranking of requirements. BACKGROUND There is an enormous literature on tradeoff analysis and decision theory. A generic process making tradeoffs consists of the following steps: 1. Decompose one’s understanding, of the overall problem to be solved, into discrete objectives or requirements (the terms are often used interchangeably) 2. Develop a performance scale for each objective. The scale might be binary (present, absent). 3. Determine the relative preference for different combinations values on the scales. A mathematical function on the objective values that expresses a stakeholder’s preferences through the function’s values goes by a variety of terms in the literature. “Figure of Merit”, “value function” and “utility function” Are all used, though with somewhat different meanings. In the formal literature a utility function possesses formal properties beyond simply capturing a stakeholder’s preferences between two deterministic alternatives. (Kreps, 1988), (Daniels and Werner, ##) 4. Evaluate the utility for a set of alternative choices of system configuration. 5. Make a decision based on the evaluated utility of the alternative choices. In the canonical case we make an exclusive decision in step #5, we select the most preferred alternative. In many realistic cases we might choose other courses of action, such as expanding our search, or rethinking the original problem because of poor performance of all known alternatives, returning to the sponsors to recommend constraints be relaxed, or make an incremental implementation decision on the elements that seem to be most certain and restart the process in a new increment of the spiral. All of these steps have been addressed in the literature, at varying levels of formalism from entirely heuristic to rigorously mathematical. Two basic references are (Keeney & Raiffa, #) and (Keeney, 1992). The first of these is a canonical textbook on the construction of utility functions and their use in decision analysis. It emphasizes the use of additive utility functions, but does not advocate the use of the common (and incorrect) subjectively weighted additive utility functions. The second of these books is closest to the subject of this paper. In (Keeney 1992) Keeney describes a wide variety of heuristics for determining objectives for a decision, creating scales of measure, and the suitability of sets of objectives for making decisions. Keeney also discusses alternatives to hard decisions in step #5. A recent SE paper (Daniels, 2001) discusses detailed issues pertinent to step #3 if one is using additive utility functions, although the terminology “Figures-of-Merit” is used there. The popular Analytical Hierarchy Process (AHP) (Saaty, 1990#1980? and Saaty, ##) takes its own perspective on steps 1 through 5. In AHP objectives are decomposed hierarchically downward with priority determined by a specific weight elicitation process. Alternatives are compared on the objectives without detailed scales, but using a specific pairwise scale unique to AHP. In place of a value or utility function there is an additive roll-up based on the pairwise comparisons. The AHP method is not formally equivalent to utility theory, a subject that has generated many papers over the years. While the decision theory literature defines the generic steps with great rigor, real-world applications virtually never reach the idea of rationality captured in decision theory. Some of the reasons for imperfect practice are themselves theoretically sound. For example, we know from a variety of psychological experiments that people’s actual behavior and preferences do not correspond precisely to the assumptions of utility theory. In addition, real decisions have to be made on deadlines, and information can never be perfect. Real decision makers are forced into the use of heuristics for choosing when to stop gathering information, how much complexity to deal with, and how to trade off risks. But, it is also observed that actual practice falls short in adopting overly simplified models and ad-hoc methods simply because well-grounded but pragmatic tools are not widely available. (Gilb, 1976 and ##) Common problems include minimal guidance selecting objectives and building scales, utility functions built on additive combination with subjective weights, and decision-making restricted to selecting a single alternative. This paper presents a set of pragmatic methods, built on years of practice, which can address the real complexities of all five steps. The paper concentrates on step #3 above, priority determination. It is built on additional guidance for step #1 and 2, building scales of measure (Gilb# CE Chapter 5), a context of evolutionary development (Gilb# CE Chapter 10, Larman and Basili 2003, MacCormack et al 2003), and the capture of engineering information in Planguage (Gilb CE 2004). Priority determination should be: • an information-based process, which makes full use of the current and continuous available factual information and, is able to reuse this information. • a dynamic process, which uses feedback from the on-going implementation, and is open to instigating and catering for changes in both requirements and design ideas. • a resource-focused process, which considers Return on Investment (ROI) and takes into account resource availability. For the purposes of this paper we assume a fairly large-scale systems engineering environment, which includes: • multiple stakeholders, each probably ignorant of the environments of some of the other stakeholders • multiple competing (for common scarce resources) requirements, from the stakeholders • (understandable) ignorance by stakeholders of the real resource costs of delivering their requirements • lack of an agreed understanding of the overall system-wide picture: both the system-wide requirements and the actual, available system-wide resources. • complex decision-making processes (including meetings and reviews), which are affected by personal management agendas • uncertainty about the future, in which resources, requirements, authority, power and technology will change. • an incremental delivery environment where we can deliver systems in increments, learn about objectives and priorities at each increment, and can (and must) feed that learning back into subsequent development. This paper demonstrates how Planguage, a specification language and set of methods, which one of the authors (Gilb) has developed over many years, has the capability to address all these above aspects.