8B-i

APPENDIX 8B. UNCERTAINTY AND VARIABILITY 

TABLE OF CONTENTS

8B.1 INTRODUCTION ...................................................................................................8B-1

8B.2 UNCERTAINTY .....................................................................................................8B-1

8B.3 VARIABILITY ........................................................................................................8B-1

8B.4 APPROACHES TO UNCERTAINTY AND VARIABILITY ...............................8B-2

8B.5 PROBABILITY ANALYSIS AND THE USE OF CRYSTAL BALL...................8B-2

LIST OF FIGURES 

Figure 8-B.5.1 Normal Probability Distribution ..................................................................8B-3
8B-1

APPENDIX 8B. UNCERTAINTY AND VARIABILITY

8B.1 INTRODUCTION

 Analysis of a potential energy efficiency standard involves calculating effects, for 

example, the effect of a standard on consumer life-cycle cost (LCC). To perform the calculation, 

the analyst must first: (1) specify the equation or model that will be used; (2) define the 

quantities in the equation or model; and (3) provide numerical values for each quantity. In the 

simplest case, the equation is unambiguous (it contains all relevant quantities and no others), 

each quantity has a single numerical value, and the calculation produces a single value. 

Unambiguousness and precision are rarely the case, however. In most cases, the model and/or 

the numerical values for each quantity in the model are not completely known (i.e., there is 

uncertainty) or the model and/or the numerical values for each quantity in the model depend 

upon other conditions (i.e., there is variability).

 Thorough analysis involves accounting for uncertainty and variability. Although the 

simplest analysis involves a single numerical value for each quantity in the calculation, 

arguments can arise about the appropriate value for each quantity. Explicit analysis of 

uncertainty and variability provides more complete information to the decision-making process.

8B.2 UNCERTAINTY

 When making observations of past events or speculating about the future, imperfect 

knowledge is the rule rather than the exception. For example, the energy consumed by a 

particular type of appliance (such as the average residential clothes washer) is not recorded 

directly, but rather estimated based on available information. Even direct laboratory 

measurements have a margin of error. When estimating numerical values expected for quantities 

at some future date, the exact outcome rarely is known. 

8B.3 VARIABILITY

Variability in the calculation of a quantity means that different applications or situations 

produce different numerical values. Specifying an exact value for a quantity may be difficult 

because the value depends on something else. For example, the number of hours a household 

operates a clothes washer depends on the specific circumstances and behaviors of the occupants 

(e.g., number of persons, personal habits). Variability makes specifying an appropriate 

population value more difficult, because no one value is likely to be representative of the entire 

population. Surveys can be helpful here, and analysis of surveys can relate the variable of 

interest (e.g., hours of use) to other variables that are better known or easier to forecast (e.g., 

number of persons per household). 
8B-2

8B.4 APPROACHES TO UNCERTAINTY AND VARIABILITY 

 This section describes two approaches to uncertainty and variability: 

• scenario analysis and 

• probability analysis. 

Scenario analysis uses a single numerical value for each quantity in a calculation, then 

changes one (or more) of the numerical values and repeats the calculation. Numerous 

calculations are performed, which provide some indication of the extent to which the result 

depends on the assumptions. For example, the LCC of an appliance could be calculated based on 

electricity costs of 2, 8, and 14 cents per kilowatt-hour. 

The advantages of scenario analysis are that each calculation is simple; a range of 

estimates is used; and crossover points can be identified. (An example of a crossover point is the 

energy rate above which the LCC is reduced, holding all other inputs constant; that is, the energy 

rate at which the consumer achieves savings in operating costs that more than compensate for the 

increased purchase price.) The disadvantage of scenario analysis is that there is no information 

about the likelihood of each scenario. 

 Probability analysis considers the probabilities within a range of values. For quantities 

characterized by variability (e.g., electricity rates in different households), surveys can be used to 

generate a frequency distribution of numerical values (e.g., the number of households subject to

electricity rates at particular levels) to estimate the probability of each value. For quantities 

characterized by uncertainty, statistical or subjective measures can be used to provide 

probabilities (e.g., manufacturing cost to improve energy efficiency to a given level may be 

estimated to be $10 ± $3). 

 The major disadvantage of the probability approach is that it requires more information, 

namely information about the shapes and magnitudes of the variability and uncertainty of each 

quantity. The advantage of the probability approach is that it provides more information about 

the outcome of the calculations; that is, it provides the probability that the outcome will be in a 

particular range. 

 Scenario and probability analysis provide some indication of the robustness of a policy 

given the identified uncertainties and variability. A policy is robust when the impacts are 

acceptable over a wide range of possible conditions. 

8B.5 PROBABILITY ANALYSIS AND THE USE OF CRYSTAL BALL

 To quantify the uncertainty and variability that exist in inputs to the engineering, LCC, 

and payback period analyses, DOE used Microsoft Excel spreadsheets combined with Crystal 

Ball, a commercially available add-in software, to conduct probability analyses. The probability 

analyses used Monte Carlo simulation and probability distributions. 
8B-3

Simulation refers to any analytical method meant to duplicate a real-life system, 

especially when other analyses are too mathematically complex or difficult to reproduce. 

Without the aid of simulation, a spreadsheet model will reveal only a single outcome, generally 

the most likely or average outcome. Spreadsheet risk analysis uses both a spreadsheet model and 

simulation to automatically analyze the effect of varying inputs on outputs of the modeled 

system. One type of spreadsheet simulation is Monte Carlo simulation, which randomly 

generates values for uncertain variables numerous times. Monte Carlo simulation was named for 

Monte Carlo, Monaco, where the primary attractions are casinos containing games of chance. 

Games of chance such as roulette wheels, dice, and slot machines, exhibit random behavior. The 

random behavior in games of chance is similar to how Monte Carlo simulation selects variable 

values at random to simulate a model. When you roll a die, you know that a 1, 2, 3, 4, 5, or 6 will 

come up, but you do not know which number for any particular roll. So too with variables that 

have a known range of values but an uncertain value for any particular time or event (e.g., 

product lifetime, discount rate, and installation cost). 

For each uncertain variable (a variable that has a range of possible values), a probability 

distribution is used to define possible values. The type of distribution selected is based on the 

conditions surrounding that variable. Types of probability distributions include the following. 

Figure 8B.5.1 Normal Probability Figure 8-B.5.2 Triangular Figure 8-B.5.3 Uniform Probability

Distribution Probability Distribution 

Distribution

During a simulation, multiple scenarios are calculated by sampling values repeatedly 

from the probability distributions for the uncertain variables. Crystal Ball simulations can consist 

of as many trials (or scenarios) as desired—hundreds or even thousands. During a single trial,

Crystal Ball randomly selects a value from the defined possibilities (the range and shape of the 

probability distribution) for each uncertain variable and then recalculates the spreadsheet. 

NORMAL TRIANGULAR UNIFORM