The NBS Energy Model

Assessment Project: Summary and

Overview

S I Gass

K. L Hoffman

R H. F Jackson

L S. Joel

P 3. Saundeis

Operations Research Division

Center for Applied Mathematics

National Engineering Laboratory

U S Department of Commerce

June 1980

Issued September 1 980

Submitted to: Office of Analysis Oversight and Access

S. Department of Energy

shington, DC 20461

national *v*xatj

OF STANDtil*

I.tHRART NOV ? 4 1980

NBSIR 80-2128 , \

THE NBS ENERGY MODEL

ASSESSMENT PROJECT: SUMMARY AND

OVERVIEW

)Q,

O

S. I. Gass

K. L. Hoffman

R. H. F. Jackson

L. S. Joel

P. B. Saunders

Operations Research Division

Center for Applied Mathematics

National Engineering Laboratory

U.S. Department of Commerce

National Bureau of Standards

June 1980

Issued September 1980

Submitted to: The Office of Analysis Oversight and Access

U.S. Department of Energy

Washington, DC 20461

U.S. DEPARTMENT OF COMMERCE, Philip M. Klutznick, Secretary

Luther H. Hodges, Jr., Deputy Secretary

Jordan J. Baruch, Assistant Secretary for Productivity, Technology, and innovation

NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director
v ' • ; >

' JAv. - . •

•» * i. .*' >
ABSTRACT

This report is a summary of the activities and technical reports for the

Energy Model Validation Procedure Development project undertaken by the

Operations Research Division for the Department of Energy, using DOE’s Midterm

Oil and Gas Supply Modeling System (MOGSM) as a test vehicle. The reports

cover: (1) assessment of the documentation of MOGSM; (2) analysis of (a) the

model methodology, (b) characteristics of the input and other supporting data,

(c) statistical procedures undergirding construction of the model, and (d)

sensitivity of the outputs to variations in input; as well as (3) guidelines

and recommendations for the role of these in model building and developing

procedures for their evaluation.

Key Words: Assessment; documentation; energy; forecasting; mathematical

models; sensitivity analysis.

I. INTRODUCTION

This report summarizes the activities of a DOE-sponsored project for "Energy

Model Validation Procedure Development". The project* s basic purpose was to

develop and apply standards and procedures for the assessment of analysis

systems (models) utilized by the Energy Information Agency of the Department

of Energy. This work, and other related DOE activities, has the goal of

developing methods for finding the degree of confidence in each system’s

results and the circumstances under which a system may be used.

To initiate this research, the DOE Midterm Oil and Gas Supply Modeling System

(MOGSM) was selected as a test vehicle for idea development and experimentation. Any proposed assessment standards and procedures were to be applied

to the latest version of MOGSM. As given In the project statement of work,

objectives of this project were: (1) to develop methods useful for validating

EIA analysis systems; (2) to establish a team of analysts consisting of NBS

personnel and outside consultants that will accumulate and maintain expertise

in validation procedures as applied to DOE analysis systems; (3) to apply proposed validation procedures to the DOE midterm oil and gas supply models; and

(4) to specify system validation standards and procedures based on the experiences of the midterm oil and gas supply model evaluation.

These objectives were to be accomplished by a set of tasks that included an

evaluation of the documentation and the identification of documentation deficiencies; evaluation of model attributes, including data, mathematical formulation, statistical and other estimation procedures; analysis of system sensitivity, system performance and computer-related system characteristics; and
2

the specification of assessment activities necessary to support the determination of confidence in a system’s results.

It should be stressed that assessment of the MOGSM was not our primary goal.

NBS’s task, as described in the statement of work, was to use the MOGSM as a

means for developing an assessment methodology for possible use by DOE and

other energy modeling groups. Model assessment as a formal discipline is In

its early stages of development. This project was designed to contribute to

this new and important "modeling" activity.

The body of this report consists of condensations of the reports produced in

the course of execution of the project tasks, followed by a summary of overall

conclusions and recommendations . Thus the report could be viewed as an

executive summary for the project.
3

II. PROJECT REPORTS

The activities associated with the development of energy model validation procedures are described in detail in the 10 reports listed below by title; complete references are given in Section V. The following subsections of this

section discuss each report according to the scheme given below, and each subsection includes a brief statement of the conclusions and recommendations resulting from the assessment activities described in the listed reports.

Section A ; Interim Report on Model Assessment Methodology: Documentation

Assessment.

Section B : An Annotated Restatement of the Midterm Oil and Gas Supply Modeling

System Methodology.

Section C : Investigation of Underlying Data: Midterm Oil and Gas Supply

Modeling System .

Section D : System Sensitivity and Stability I: Model Validation, Simulation , and Sensitivity Analysis .

System Sensitivity and Stability II: A Statistical Method for the

Assessment of Model Sensitivity to Input Variables.

Data Extrapolation and Statistical Forecasting .

The Role of the Statistician in Energy Model Development and Validation.

Section E : Sensitivity Analysis of DOE Forecasts of Midterm Oil and Gas Supply

for the 1978 Annual Report to Congress.

Section F ; Concepts of Model Confidence.

Section G ; Conversion Costs: A Practical Exercise in Model Portability.
4

A. Interim Report on Model Assessment Methodology: Documentation Assessment ^

We approached the task of evaluating the MOGSM documentation from the viewpoint of model assessors seeking answers to the following questions:

1. What was the model supposed to be?

(Documentation accompanying the model is the only proper source of

such information.)

2. What did the model turn out to be?

(The computer code is a necessary, but not sufficient source for

this information.)

3. Is the resulting form consistent with the intent?

We proposed to answer these questions by obtaining a sufficient understanding

of the. conceptual model and computer code to enable us to run the computer

system under different test scenarios. W

Te discovered early that the HOGSM

documents available from DOE did not contain sufficient information to pursue

this course directly. The MBS project staff was able to augment the MOGSM

documentation by locating a number of ancillary documents. With the support

of the original model developers and DOE staff, we were then able to develop

an understanding of the conceptual model and its computer realization sufficient to permit continuation of the project. Without this very time consuming

and painstaking effort, the project could not have continued.

Our initial approach to assessing the available MOGSM documents was to examine

them in terras of previously proposed model documentation guidelines^ to

*See reference [1].

^The guidelines used were as presented in Gass, S. I., "Computer Model Documentation: A Review and An Approach," NB5 Special Publication 500-39,

National Bureau of Standards, Washington, D. C. , February 1979.
5

determine whether documents that conform to these proposed guidelines were

sufficient for the needs of model assessors. However, the information in the

DOE documentation was not close enough to the information requirements of the

guidelines for this comparison to be accomplished. We therefore developed a

revised set of model documentation guidelines, based on a review of the MOGSM

documentation. The project report presents the revised guidelines and

discusses (a) the denciencies of the available MOGSM documents and (b) how

MOGSM documents that conformed to these guidelines would have benefited DOE.

The documentation guidelines proposed in the report represent a hierarchial

approach to information requirements in which the four categories of document

types are associated with phases or levels of model use. A higher level

includes all of the documents of a lower level. The levels and brief

descriptions of the required documents follow.

0

Level I: Rote Operation of the Model

Level I is concerned with requirements for rote execution of computer runs, i.

e. the "ground rules for setting up and running the model" on a particular

computer and for verifying the correctness of the execution. The document

types are:

o Operations Manual—provides computer operations personnel with a description of the software and operational environment so that the

software can be run;

o Data Base Description: Physical/Logical Characteristics —presents instructions for data entry, tape and file labeling conventions, support

software descriptions, logical data characteristics (e.g., arrangement,

relationships), and physical characteristics (e.g., storage, access);
6

o Software Description: User Level —describes the functions performed

by the software in non-ADP terminology, so that the user organization

can determine how to apply and operate the model.

Level II: Model Use

Level II involves an explanation of a given set of scenarios and enables

construction of new scenarios and interpretation of output. The relevant

documents are those specified in Level I and:

o Mathematical Description—describes the complete details of the

mathematical/ logical model, assumptions and hypotheses, rationale for

the form of the model, discussion of alternatives, and restrictions

on model use;

o Data Requirements Report: Sources, Transformations, and Justification—describes the detailed data needs of the model including both

input variables and "hardwired" parameters;

o Process Description—describes the underlying physical, economic,

technological, and behavioral processes to be modeled for readers

unfamiliar with the topic area.

Level III: Model Maintenance

In this level, the documentation addresses modification of the computer code

(and perhaps, therefore, the conceptual model) to investigate scenarios which

range beyond originally conceived limits or assumptions. The relevant documents are those specified in Levels I and II and:

o Software Description: Programmer Level—provides a maintenance pro

grammer with the information necessary to understand the programs,

their operating environment, and their maintenance procedures;

Maintenance Log—records changes made to the models and/ or its data

in a manner which facilitates the extraction of an audit trail.

o
7

Level IV: Model Assessment :

In order to conduct a third-party assessment, model assessors should have all

model documentation available to them. For a variety of reasons, the report

types listed below may not have been produced, but if they do exist they

should be made available to the assessment team.

The relevant documents are those specified in Levels I, II, and III and:

o Assessment Report —describes any model assessment plan agreed to by the

user/sponsor and model developer, and the results of implementing that

plan;

o Model Application Report—describes for decision makers the results of

exercising the model to answer specific questions or to study the behavior of the problem environment as represented by the model;

o Model Summary—presents a concise nontechnical description of the model

so that a broad audience can determine If it is of interest to them;

o Historical Record—describes the questions or problems that led to the

decision that the model was needed, and how the model is to be used to

address these issues;

o all other documents written about the model and not specifically

listed above.

This report stresses that model documentation should not be judged on the

basis of whether the designated documents exist, but rather by whether or not

all of the information required in each document is readily available in a

well-organized manner.
8

?>,e commendations

Specific recommendations to DOE are the following:

o The four-level documentation guidelines should be adopted provisionally by DOE and applied to future modeling activities.

o Model documentation should proceed pari passu with model development

to improve efficiency of production and quality of documentation.

o To improve current deficiencies in DOE model documentation, the report identifies those information requirements that are sufficiently crucial

to continuing model use and necessary for model assessment to warrant post hoc

preparation.

o A model assessment team should have at least oae member who has no

initial knowledge of the model.

B. An Annotated Restatement of the Midterm Oil and Gas Supply Modeling System

I Methodology ^

The NBS project to develop methodologies for the assessment of energy models

began with an assessment of the MOGSM documentation. One conclusion of the

documentation assessment was that the existing documentation did not contain

information sufficient for us to obtain a thorough understanding of the mathematical/logical structure of the model. However, through many discussions

with the model developers, the structure of the model was exposed and the model assessment effort continued. This NBS report arose, then, through our recording for our own use of our interpretation of the mathematical/logical

structure of the model. It was pieced together from portions of the original

documentation, from discussions with the developers, and our analysis.

^See reference [9].
9

Since our major concern was the development of assessment methodology, we did

not intend to supplement the "primary" documentation of the model we were

assessing. However, the need for at least an internal document became

apparent. As the production of this document progressed, we realized that

this activity was advancing our general methodology-development goals. It

clarified and synthesized many of our notions about documentation that were

incorporated into the guidelines reviewed in Section A of this summary report.

We also believe that the existence of this report will help advance the most

important by-products of proper model assessment—making this model more

"transparent", i. e. , more open to peer review and more understandable to

policy makers who rely on information produced by models.

Our restatement report describes the methodology used by the Department of Energy (DOE) to estimate oil and natural gas supply curves for use by the midterm energy forecast system. It first describes the key features of the oil

and natural gas supply process modeled in MOGSM. It next describes, in

general terms, the model used to project future oil supply as a function of

federal and corporate policy actions and market prices for crude. Finally, it

discusses the estimation of natural gas supply, highlighting the differences

between it and the oil supply estimation.

A number of points need to be made concerning this assessment activity. MOGSM

consists of three submodels: the Economic Submodel, the Resource Base Submodel, and the Drilling Submodel. Descriptions of both the Economic Submodel

and the Resource Base Submodel are based on the verbal descriptions provided

by the model developers. No computer code listings were examined to see if

there were discrepancies between the structure described by the modelers and
10

the computer implementation. We have, however, requested the model developers

to review a draft of this document to ensure that what we have written is consistent with their "conceptual" model.

Information on the Drilling Submodel was obtained by reading the computer code

labeled "OILDRL 78”. This exercise uncovered discrepancies between the computer implementation and descriptions in either the existing documentation or

in verbal statements made by the developers. These discrepancies are noted in

the report.

A major portion of the model and its methodology was totally undocumented at

the outset of the present exercise. Thus, the methodology used to determine

the costs of exploratory and developmental drilling are presented for the

first time in this document, thereby allowing review of that submodel.

It is a difficult task to document work performed by others and to record verbal explanations of a highly complex model. Furthermore, we do not believe

that the task is complete until the computer code of both the Economic Submodel and the Resource Base Submodel have been reviewed by a third party to uncover discrepancies between the implementation and the "conceptual" model documented.
11

Examination of the MOGSM methodology revealed questionable aspects of portions

of the model; tacit assumptions or outright discrepancies between stated process relationships and computer program equations. These include the following:

o The time order of discovery of oil pools is strictly related to size,

i.e., probability estimates about the resource base are translated

into deterministic equations in the model.

o In some cases, the magnitudes of finding rates are based on only one or two data points.

o Exploratory equipment (rigs) are double counted in projecting drilling

requirements.

o Finding rate estimates used in conjunction with the factors for

ultimate recovery, known as Hubbert multipliers, result in double

counting of some oil production.

o Planned and realized depreciation are determined by inconsistent

calculations.

o The model is configured to treat 14 regions in some calculations but

only 12 in others.

o For offshore regions a constant production rate is assumed in the

economic submodel while a nonlinear production function is assumed

within the resource submodel.

These problems have been brought to the attention of the model developers, resulting in some modifications of the computer programs (which have not been

reassessed). The sensitivity of model outputs to the identified anomalies has

not yet been determined completely, although some of the sensitivity analysis

discussed later in Section E of this report bears on the finding rate question

and the resource base estimates.
12

Recommendations

Some conclusions and recommendations relating to assessment methodology deriving from the preparation of this report are:

o An assessment report is a useful part of the extended documentation of

a model. It should record model anomalies or deficiencies.

o Assessment documentation must conform to standards required of the rest

of model documentation.

o A description of a process being modeled is essential for model assessment. Such a description must exist and be referenced in the methodology documentation. It serves as a safeguard against misinterpretation in the application of a model.

o The only sure source for identification of discrepancies between a conceptual and a realized model is the computer program. Someone must

be prepared to read code in the process of assessment. Third-party

assessment of computer code can uncover coding inconsistencies (unsuspected by the model builders) in different portions of a model.

o Assessment findings and assessor-produced additional model documentation should be reviewed by the model developers. If possible, the

modelers' responses should be incorporated into the assessment report.

o The statement that the process of documentation can uncover weaknesses

in one's understanding of the mechanics or the assumptions of a model

applies to model assessors as well as model developers.

C. Investigation of Underlying Data: Midterm Oil and Gas Supply Modeling

System ^-

This report presents the findings of the task to investigate, using existing

data documentation, the sufficiency of the MOGSM's underlying data. The objective of the study was to define and achieve a state of “audit readiness"

for MOGSM's data, not to perform a data audit or validation study. A state of

"audit readiness” is defined to be that state of an analysis system's input

data and source data which permits an audit on that data without further assembly preparation, or collection. A data audit is a study to determine the

1-See reference [ 3 ].
13

appropriateness, accuracy, timeliness, and completeness of the data. The

study was achieved by activities described by the following subtasks:

1. Summarize DOE's data documentation of the model's input data (both

internal and external to the computer programs that implement the model).

2. Provide a complete set of the input data operated on by the model,

including data elements embedded in the coding.

3. Provide a complete set of the source documents containing the raw

data from which the model's input data are derived.

4. Specify the intermediate analysis and processing steps by which raw

data are transformed into input data.

5. Identify possible alternative sources of raw data.

6. List errors found in the raw data or the input data, including any

discrepancies between the purpose for which the raw data were originally

gathered and the purpose for which they are used in the model. (This task was

specifically not intended to include any particularly expensive research

effort to answer this question.)

7. Develop a precis of the input data and data documentation. This precis is to enable EIA's Office of Energy Information Validation, to audit the

data
14

The final report contains three major sections: the data precis , the input

data listings, and the summary of data transformations. The presentation of

data in this fashion represents a tentative approach for describing an analysis system’s input data.

Recommendations

With respect to a model's data, generic findings about assessment include:

o The data associated with an analysis system should be regarded as part

of that system. Specifically, an analysis system should always include an audit of the data associated with the system,

o The assessment performed under contract to NBS included a description

of the method used to achieve "audit readiness.” Since no audit took

place, a methodological approach for auditing large data sets was not

developed. This next stage of data assessment is critical and should

be undertaken.

o The subsequent process of auditing an analysis system's input data has

the two distinct components of addressing the source data and the data

transformations. These components may require different skill mixes

in the audit team.

o An analysis system's overall design—including its data structure and

implementing programs—has a significant impact on the effort required

for assessment. The needs of model assessment should, therefore, be

reflected in the designs of future systems.
15

The report also presents two specific conclusions about the data used in

MOGSM:

o Within the limits of this study, we found no errors in the MOGSM input

data set* We did, however, note a number of inconsistencies between

original purposes of certain source data and their uses in MOGSM.

o Many of MOGSM data elements, especially those related to the economics

of exploration and production, come from a few sources. Many have

a weak statistical basis and/or rely significantly on human judgment.

However, we did not locate any acceptable alternatives to these

sources.

D. Statistical Aspects of Model Assessment *

Several studies were carried out to address the appropriateness of statistical

estimation methodologies of MOGSM and the sensitivity of model results associated with the particular parameter values specified in MOGSM. One of our

tasks was to determine the adequacy of methods used to derive parameter values

contained in the model’s mathematical representation. Particular emphasis was

to be placed upon an examination of the rationale for the estimation procedures selected for the models and of alternative possibilities. Another task

called for a sensitivity assessment of the model by comparison of outputs generated by alternative values to key model input parameters. Particular emphasis was to be placed upon an evaluation of model sensitivity and stability

relative to such alternatives.

*See references [3, 4, 5, 6].
16

Appropriate strategies for making reliable forecasts from historical data are

not known. However, there are in fact well-established statistical techniques

for projecting data trends into the future. These usually fall into the subject areas of "Forecasting" and "Time-Series Analysis." The scope of this

work is confined to what might be called short and mid-range forecasting

methods, since these are the horizons most relevant to the supply models under

assessment.

In the study reports, the more important techniques are described in detail,

in addition to some of the subjective aspects of forecast model development.

Included are: a brief discussion of the general nature of such models and

their possible application in energy analysis; a fairly formal definition of

the basic forecasting problem; the major alternative methods; performance

criteria; and energy applications. A key finding here is that the technique

for calculating the statistical uncertainty in the supply projections of the

MOGSM is inadequate. To correct this, strategies are indicated for passing

alternative scenarios through the models in a statistically correct and more

meaningful manner. The major established technique for doing this is called

model sampling. The complete details of this approach are provided, along

with a discussion of its potential use in energy modeling.

Recommendations

Based on the statistical and sensitivity analysis tasks, the following conclusions and comments are offered. Although these are substantially just common

sense, they have been emphasized here because they are frequently overlooked

in the press of analysis in response to tight deadlines.
17

o Statistical assumptions should always be stated clearly. Their

plausibility and their relevance to the requirements of the statistical methods used should be assessed. For example, the time period

from which the statistical relationship is derived for short-term

forecasting should be relevant to the period for which the forecast is

being made. In other words, forecasts from historical data embody an

assumption that the basic, underlying conditions will continue to be

in force for the prediction period.

o Short-term forecasts made by these methods usually should be combined

with, or weighed against, forecasts made independently by other methods before a final forecast is determined. Other statistical techniques may be utilized in making complementary independent forecasts.

In brief, it is dangerous to use regression analysis for forecasting

whenever the value of the independent variable(s) falls considerably

outside the range of prior data. Regression analysis is a very useful

tool if used within the bounds of its theory.

o The greatest single hazard in forecasting is that events that were unforeseen and unpredicted by the forecaster will occur to "overturn" a

forecast. Thus, the possibility of serious error in a forecast increases, as a rule,- with the length of the forecast. A forecast covering a period six months in the future is less likely to be in error

because of the impact of unforeseen events than a comparable forecast

projected five years ahead. Similarly, forecasts made during periods

of relative stability are more likely to be accurate than forecasts

made during periods when unknown and unpredictable conditions may be

encountered
18

o The techniques of time-series analysis examined in this report have

proved to be useful tools for forecasting. It should be recognized at

the start, however, that these techniques cannot be applied mechanistically; rather, subjective evaluations must play a key role in their

use. There are so many possible models from which to choose that care

must be exercised to prevent unnecessary and uncalled for detail.

There are several reasons why subjective considerations are important

in time-series analysis. First, a generally satisfactory probability

approach to such analysis has not yet been found. Second, even if a

suitable probability approach to time-series analysis were available,

some purely subjective evaluations would be necessary in making forecasts. Whenever we examine the past to obtain clues about the future

(as in the case when we use time-series analysis as an aid to forecasting) the past is relevant only to the extent that causal condi-

* tions previously in effect continue to hold in the period ahead.

Causal conditions seldom remain constant from period to period, but

rather tend to be constantly shifting. Thus, the connection between

the past, present, and the future must be continually evaluated.

E. Sensitivity Analysis of DOE Forecasts of Midterm Oil and Gas Supply for

the 1978 Annual Report to Congress ^

The objective of this investigation was to assess the quality and usefulness

of the midterm projections of oil and gas production given in Volume III of

EIA’s 1978 Annual Report to Congress (ARC). The methodology to be used was to

be based on the knowledge, methods, and results from NBS’ previous validation

*See reference [7].
19

work. The study design included a set of statistical experiments to analyze

the sensitivity of MOGSM outputs to systematic variations of inputs, and a review of some of the MOGSM methodology and its impact on results, possibly in

comparison with other published forecasts.

Specifically, our statistical experiments included: (1) reestimation of finding rates* by various statistical methods applied to historical data on (regional) exploratory drilling and the corresponding oil and gas reserve additions; (2) analysis of the effects of Monte Carlo variations of regional

resource base estimates on finding rates and on MOGSM results; and (3) a

"response surface" analysis (described below) to identify those input data

elements whose variations have the greatest impact on MOGSM outputs. These

experiments were to be supplemented by a Monte Carlo analysis of the effects

on MOGSM output of variation in the critical variables identified in the

response surface experiment, but this step was aborted owing to resource limitations. Briefly, we found that:

o In Experiment 1, different methods of estimation yield substantially

different values of regional oil and gas finding rates. When applied

to MOGSM, the various finding rate estimates in turn lead to forcasts

of a wide range of oil and gas production, as shown in the following

table.

Regression Regression

Method of Determination Analysis Time Series Analysisof Finding Rate: *78 ARC Met Analysis Alter. Meth.

Crude Oil (MM Bbl/day) 6,905 6,510 5,520

Natural Gas (TCF/yr) 14,476 15,599 10,311

NGL (MMBbl/day) 1,152 1,149 840

(average values over the 1985-2000 period)

*As used in the model, the finding rate is the amount of proved reserves added

per foot of exploratory drilling.
20

o Statistical considerations do not indicate a preferred estimation

method or corresponding (set of) production forecast(s) in this

instance. The finding rate estimates actually employed in generating

the 1978 ARC forecasts appear to be as reasonable as any of the alternatives. But, the associated uncertainty is high, as the above ranges

suggest.

o In Experiment 2, treating the undiscovered regional resource bases

(oil and gas) as random variables via the Monte Carlo analysis leads

to uniformly and significantly higher oil and gas production forecasts

from MOGSM than does the current (deterministic) procedure. Currently, MOGSM operates with specific regional resource base estimates.

For example, the Series C forecast in the 1978 ARC is based upon the

50^ percentile or median values in the resource base estimates set

forth in USGS Circular 725. The Monte Carlo analysis instead treated

resource base estimates as log-normally distributed random variables

(consistent with USGS Circular 725). The set of MOGSM-generated oil

and gas production forecasts resulting from the analysis shows these

properties, relative to the ARC forecasts:

- The medians of the Monte Carlo forecasts are all markedly more

"optimistic" than the 1978 ARC Series C medium geology forecast,

as shown in the following table.
21

RESOURCE BASE PRODUCTION

ESTIMATES

1985 1990 1995

Crude Oil (MM/Bbl/day)

ARC Series C 6,903 7,081 7,013

Monte Carlo Median 7,739 8,819 9,338

Natural Gas (TCF/yr)

ARC Series C 16,777 15,595 11,802

Monte Carlo Median 21,754 22,154 19,828

NGL (MM/Bbl/day)

ARC Series C 1,338 1,218 1,092

Monte Carlo Median 1,586 1,686 1,539

- The range of the Monte Carlo forecast production levels is uniformly

higher than the ARC forecast production levels. For example, the

median Monte Carlo crude oil forecast is comparable with the highest

ARC forecast.

- The spread between the 5

C^ percentile and 95 tn percentile of the set

of Monte Carlo forecasts is less than the difference between the

median of the Monte Carlo forecasts and the ARC Series C forecast.

That is, the MOGSM forecasts of oil and gas production are less

sensitive to the inherent randomness in regional resource bases than

they are to the analytical methods by which resource base estimates

are applied to MOGSM.

In general. Experiments 1 and 2 show that MOGSM oil and gas production forecasts are highly sensitive both to inherent variations in certain MOGSM input

data elements and (even more) to the analytical procedures used in defining or

estimating these elements.
22

Experiment 3, response surface analysis'*- , was carried out in two stages.

Variables judged to be of strong influence were divided into four classes:

geologic (finding rates), economic (price, discount rate, various costs), geologic/technological (decline rate, recovery factors), and drilling/operating

(lease acreage, equipment planning, and attrition). These furnished a basis

for four factorial design "subexperiments" on two values of the variables

("high" and "low", with central or "base" values omitted), each of which

yielded a linear response model. Then, variables from all the classes showing

the greatest influence in the first stage, were combined to construct an

"integrated" response surface.

All response surface equations emerging from the first stage experiments show

high R“ values (>0.90 in all cases) indicating that the model is wellapproximated by these linear forms. By way of verification, the equations all

closely fit the corresponding center points (which were not used in deriving

the equations, but reserved for calibration purposes). However, the equations

o which define the integrated response surface do not show particularly high Revalues (oil: .88, gas: .64), suggesting that the true response surface

requires a nonlinear approximation. Time did not permit us to undertake an

investigation into possible nonlinear forms.

^-Response surface analysis involves development of a simplified model of a system relating its inputs to its outputs. Until recently, the technique was applied to physical experiments. Here, it has been useful for sensitivity

analysis of complicated, large-scale, mathematical models.
23

Perhaps the dost striking aspect of this analysis is the prominence of "economic” data elements in the set of critical elements. By contrast, previous

discussion of MOGSM’s sensitivity to input data (EIA 1978) focused on drilling

and geology-related data and ignored economic data. Similarly, the 1978 ARC

itself emphasized possible changes in physical phenomena (e.g., the "high geology" and "low geology" scenarios) more than possible changes in prices and

costs.

In summation, two sets of input data elements are critical to MOGSM ’ s forecasts, on the basis of this experiment:

- the regional resource realizations, and through them, the

regional finding rates, and

- the economic elements appearing in the table below.

We have not had the opportunity to investigate the relative importance of

these two classes of data, but such an investigation involves only a modest

extension of the present study, and would be most useful If performed.
24

SENSITIVITY OF OIL AND GAS PRODUCT MEASURES TO CHANGES

IN SELECTED INPUT DATA ELEMENTS

(Integrating Response Surface)

LEADS TO THESE CHANGES IN

OIL PRODUCT MEASURE & GAS

FOR THIS MOGSMS

INPUT ELEMENT

THIS AMOUNT OF

CHANGE

PRODUCT MEAS URE

(in 10 3 Bbl/day) (in 10 3 BC)

Price $10/Dbi( 1 ) $i/mcf( 1 ) 834.2

129.6

Discount Rate 5%(1) -315.4 -808.2

Royalty Rate io%C 1 ) -232.1 -216.7

Drilling Costs 20%( 2 ) -262.3 0

Primary Recovery Factor 10%( 2 ) 569.5 0

Total-to-Exploratory

Drilling Ratio

10%(2) -152.3 477.9

Notes

(1) These are changes in numerical units

"dollars" or "percentage points".

( 2 ) These are percentage changes from nominal or base values.
25

Recommendations

The various sensitivity analysis experiments and their results lead to these

conclusions

o The estimation for MOGSMS of regional oil and gas finding rates from

historical time series should, in the future, employ time series

analysis (second-order exponential smoothing was demonstrated in Experiment 1) instead of regression methods. Time series analysis was

found to be more robust, more flexible in its fitting of the historical data. Moreover, time series methods (by their very nature) do not

require independent estimates of regional resource realizations, which

are essential for the present regression analysis approach. Indeed,

the time series analysis itself produces independent estimates of regional resource realizations (from purely statistical treatment of the

regional time series on reserve additions and cumulative drilling

footage). The independent estimates obtained in this experiment correlate well with the mean regional resource estimates set forth in

USGS 725.

Consequently, MOGSMS projections can be decoupled from the uncertainties in regional resource estimates. The use of time series analysis

in this application removes any need in MOGSMS for regional oil and

gas resource estimates (which are, by their nature, highly uncertain)

and creates an independent, statistically based source of estimates

useful for other EIA purposes.
26

Another property of time series analysis further recommends its use in

this application. Time series analysis weighs the recent years more

heavily than the early years in the time series. Regression analysis

gives equal weight to all of the years. Because time series analysis

focuses on those years that have witnessed the greatest changes in

technoeconomic factors and are likely to be most relevant to the future, it carries not only greater statistical meaning than regression

analysis, but also greater process meaning. (We are pleased to note

that EIA has adopted time series analysis for the 1979 ARC work involving HOGSMS, as a consequence of this study.)

o The analytic procedures demonstrated in this study lead to HOGSMS

forecasts of midterm natural gas production that are significantly

higher (say 7 1/2 percent to 15 percent) than those shown in the 1978

ARC for a given set of assumptions (e.g., the Series C scenario). The

reestimation of regional finding rates via time series analysis (Experiment 1) leads to an increase of 10.0 percent in the MOGSMS forecast of average gas production over the period 1985-1990-1995. Similarly, the treatment of uncertainty in regional resource base estimates via the Monte Carlo analysis (Experiment 2) leads an increase of

7.7 percent in the HOGSMS forecast of gas production.
27

The regional finding rates estimated via time series analysis in Experiment 1 are themselves subject to some dispersion or uncertainty.

One could, therefore, carry out a Monte Carlo analysis (similar to

that in Experiment 2) to explore the effect of that variance on the

MOGSMS forecasts of gas production. Were such an experiment performed, it would likely yield a gas production forecast higher than obtained in either Experiment 1 or Experiment 2. (Indeed, the resulting

increase in the gas production forecast might approach the sum of the

indicated increases in Experiments 1 and 2.)

A 10 percent increase in projected gas production over the midterm

period is significant. It amounts to 1.5 Quads/year over the entire

midterm period—comparable, for example, to the low range of the solar

energy contribution forecast for 1995-2000 in the recent Domestic

Policy Review on Solar Energy.

o EIA should explore the prospects and benefits of a continuing program

to characterize uncertainty and treat its effects in various DOE analysis systems. This study can be considered as a pilot demonstration

of some of the practical statistical methods that can be employed in

an operational program. It also illustrates the nature of the results

that can be anticipated from such a program. In this study, sensitivity analysis led to shifts in the nominal values of certain forecasts,

reassessments of the "criticality" of certain data elements, and to

sharpen definition of the associated uncertainty. For example, the
28

Monte Carlo analysis reduced by a factor of 10 the variation in oil

and gas production projections stemming from uncertainty in a key input element (regional resource estimates).

o Uncertainty, randomness, or measurement errors in basic data need not —a priori—preclude the use of quantitative methods for policy analysis. Good analysis has a role in policy making, even in the face of

"bad" data. For example. Experiment 2 showed that (within the framework of the MOGSMS analysis) the inherent uncertainty in the ultimate

magnitude of the oil and gas resource base does not lead to a corresponding range of uncertainty in forecasts of future oil and gas production. This finding resulted from confronting and treating the uncertainty in the data, not from ignoring it.

F . Concepts of Model Confidence ^-

One of the goals of the N3S project for "Energy Model Validation Procedure

Development” is that of developing methods for finding the degree of confidence in results obtained from different analysis systems and to identify the

circumstances under which the systems may be used. The concept of model confidence is not clearly defined or understood. There is a tendency to equate

model confidence with model validity, as validity has been the main modeling

activity that leads to the acceptability and use of physically based models.

However, policy analysis models and their use in decision making requires an

extended definition of model confidence that includes validity and other

measures of model utility, i. e. those characteristics of a model that make it

*See reference [2].
29

useful and usable to a decision maker. Thus, the main effort of the project

was an attempt to define model confidence. To this end we undertook the

following activities:

1. Using the project team’s validation and assessment experiences, we

developed an initial set of confidence criteria.

2. We prepared a discussion paper on model confidence and held a workshop to (a) define model confidence, (b) review current research relevant to

the concept of model confidence, (c) discuss a preliminary methodology to be

used to measure confidence, and (d) indicate areas of future research.

3. We conducted an informal survey to obtain other opinions and/or significant issues related to model confidence.

Based on these efforts, we proposed the following initial set of confidence

criteria for a model.

o Model Definition—the problem and model environments. The information

gathered here should enable the decision maker to determine if the

problem area in question is at least within the scope of the model purposes.

o Model Structure—the theoretical and methodological bases of the model.

The information gathered here should enable the decision maker to determine if the model structure has limitations that preclude its use

as a decision aid for the problem area in question.
30

o Model Data—the data base, data sources, and procedures for data transformations. The information gathered here should enable the decision

maker to determine if data for the problem area in question are available at reasonable cost, are accurate enough, and are used correctly

by the model.

o Computer Model (Program) Verification—the tests and procedures used to

debug the subprograms and program, and how the consistency between the

program and model's mathematical and logical description was established. The information gathered here should enable the decision

maker to determine if the computer program is reliable and if it appears to be an acceptable representation for the model,

o Model Validation—The methods with which the computer model has been

analyzed in terms of its ability to produce results that can be relied

upon by the decision maker. The information gathered here should enable the decision maker to determine that the model's real-world approximation is suitable for the problem area in question,

o Model Usability—The resources, procedures, documentation, accessibility, transferability, and maintenance aspects of the model. The information gathered here should enable the decision maker to determine

if the model can be used within the decision maker's problem environment.

o Model Demographics —the model antecedents, orginators, and developers,

past users, abstract, cost, and current developmental activities.

This information should enable the decision maker to determine the

model's status with respect to past achievements, theoretical and

methodological state-of-the-art, and the expert advice that went into

its development.
31

Given such criteria, the key problem is how to measure the levels of satisfaction for each criteria. Our approach requires the decision maker to ascribe

one of five distinct levels of satisfaction for each criterion. Based on the

information gathered on the criterion, this produces a subjective but quantitative measure of the personal confidence in a model. This approach has not

been field tested as yet.

Recommendations

1. DOE should continue work in model confidence by sponsoring a task

that develops criteria and related statements from the perspective of DOE and

other government decision makers.

2. A parallel effort should experiment with the organization of materials from a DOE model assessment project into sets of information that can be

used by a decision maker to measure the seven criteria and test the proposed

confidence methodology.

G. Conversion Costs: A Practical Exercise in Model Portability

Experimental attempts to set up and run the Midterm Oil and Gas Supply Model

at computer installations other than DOE's 370/168 were planned for the purpose of development of a measure of computer model portability and for testing

the MOGSM operating documentation. The initial test plan specified a factorial design involving various computer installations and two levels of

*See reference [10].
32

familiarity with the new computer system and the model by the persons responsible for the transfer. Because inadequacies of the documentation delayed

completion of what would have served as a partial baseline case (operation of

the model by the NBS assessment team on the DOE "home” computer), only one

such experiment has been completed: transfer of the MOGSM to the UNIVAC 1108

at NBS by a programmer/analyst familiar with the model and the source computer, but unfamiliar with the UNIVAC 1108 and the NBS operating system. A successful transfer was made using the base-case scenario. The report identifies

various hardware and operating system-related obstacles to portability, primarily input/output anomalies, as well as a previously undetected error in one

of the model's subroutines.

Recommendations

The report includes several recommendations for preparation of computer programs that implement large scale models, intended to facilitate transfer of

models among computer environments. The two most general of these are:

o Use ANSI FORTRAN;

o Make the system configuration as simple as possible, avoiding large numbers of interfaces which require data transfers between modules.

The report provides in an appendix a tabulation of costs incurred in dollars

and man hours, for various phases of the transfer. These are supplemented by

a description of the contractor's knowledge of the model and the two computer

systems involved, as well as his general background, to furnish a basis for

interpretation of these costs.
33

Because tlie report was written to NBS, it is directed at an audience presumably expert in computer models in general and in the UN1VAC 1108 computer and

operating system in particular.
34

III. ADDITIONAL ACTIVITIES

In conjunction with the basic NBS/DOE project, NBS organized for DOE two symposia. The first. Validation and Assessment Issues of Energy Models , was held

on January 10-11, 1979. A proceedings volume was published with that title.

It can be ordered from the Superintendent of Documents as NBS Special Publication 569, Saul I. Gass, Editor, Stock No. 003-003-02155-5, February 1980. The

material in this 560-page volume represents a rather complete statement of the

field at that date.

The second symposium, Validation and Assessment of Energy Models , was held May

19-21, 1980. This meeting updated the state-of-the-art knowledge and attempted to extend such knowledge by having workshops in the areas of (1) validating

composite models, (2) the measurement of model confidence, (3) model standards

to aid assessment, (4) sensitivity and statistical analysis of models, and (5)

model assessment methodologies. The proceedings of this conference is in

preparation.
35

IV. SUMMARY

The activities summarized in this report are not to be construed as representing a thorough assessment of an analysis system. That was not the purpose of

the project. Our efforts exemplify approaches for addressing various aspects

of the assessment of the DOE MOGSM system and similar models. More work could

be done under each of the reported nativities and additional activities* could

be undertaken, e. g. a comparison of model results to known outcomes. In addition, there are other approaches which could be taken and other analysis

systems which could be assessed. Procedures must be developed that translate

the vast amount of information gathered during an assessment effort to statements that permit a user to determine if the results produced by an analysis

system can be utilized.

It is recommended that DOE and other sponsors, developers, and users of analysis systems initiate and/or continue to include assessment as an integral part

of the development and use of an analysis system. It is no longer acceptable

to use an analysis system which has not been the subject of assessment, either

by the modelers themselves (first party assessment) or by an independent third

party assessment team.

-We were especially disappointed that project time and funds did not permit a critical analysis of the structure of the MOGSM. We feel strongly that such

an analysis would be most beneficial to both the advancement of model

assessment methodology and the improvement of the MOGSM.
36

Although it may be argued that most of the tasks described as assessment

activities might seem to be those that should be done in the course of model

development, i. e. merely "good modeling practice,” a more careful examination

of assessment methodology indicates that it goes beyond that characterization.

For some areas of model evaluation no methodological procedures exist (e. g.

,

determining the appropriateness and consistency of the level of detail and aggregation of a model); while for others, the cost of using current methodology

may prove prohibitive (e. g., performing a sensitivity analysis of all input

parameters using monte-carlo techniques). This NBS effort and other similar

efforts are early steps in defining and advancing model assessment methodology.

The NBS staff will concentrate on two objectives in future assessment work.

One is with regard to the "ex post" accuracy of model results. Here we intend

to define and develop a measure that can be used in comparing model outputs to

historical data. The second area is an extension of the work reported in [7]

and described in section E. We intend to investigate techniques for incorporating correlations among input variables into the sensitivity analysis experiment. Throughout, we will strive to achieve a useful level of communication

with the community of non-technical model users.
37

This last issue, communicating with the non-technical users, is an important

one. At the Model Validation Symposium held at the National Bureau of Standards on May 19-21, 1980, this issue was discussed frequently. While it

seemed that progress has been made in identifying valuable techniques in model

assessment, not much effort had been expended in responding to the needs and

desires of policy makers, nor in learning how to communicate the significance

of assessment to them.

We conclude this report by stating that even as the development of model assessment methodology advances, it is not intended to lead to the certification

of models as "valid." It has (unfortunately we believe) been claimed by many

in the energy modeling community that the purpose of evaluation is certification. We believe that more attainable and useful goals for model evaluation

are to enhance system understanding, to provide directions for model improvement, and to increase credibility and confidence in energy modeling efforts.
38

V. REFERENCES

1. Gass, Saul I., Hoffman, Karla L. , Jackson, Richard H. F. , Joel, Lambert

S. , and Saunders, Patsy B. , Interim Report on Model Assessment Method- ology: Documentation Assessment , NBSIR 80-1971, National Bureau of

Standards, Washington, D. C. , January 1980.

2. Gass, Saul I. and Joel, Lambert S. , Concepts of Model Confidence , National

Bureau of Standards, Washington, D. C., to be published.

3. Harris, Carl M. , System Sensitivity and Stability I: Model Validation , Simulation, and Sensitivity Analysis , Carl M. Harris Associates,

Washington, D. C. , January 1980.

4. Harris, Carl M. , System Sensitivity and Stability II: A Statistical

Method for the Assessment of Model Sensitivity to Input Variables , Carl

M. Harris Associates, Washington, D. C. , January 1980.

5. Harris, Carl M. , Data Extrapolation and Statistical Forecasting , Carl M.

Harris Associates, Washington, D. C. , January 1980.

6. Harris, Carl M. , The Role of the Statistician in Energy Model Development

and Validation , Carl M. Harris Associates, Washington, D. C. , March 1980.

7. Harris, Carl M. and Hirshfeld, David S., Sensitivity Analysis of DOS Forecasts of Midterm Oil and Gas Supply for the 1978 Annual Report to

Congress , Carl M. Harris Associates, Washington, D. C. and David S.

Hirshfeld Associates, Inc., Potomac, Maryland, March 1980.

8. Hirshfeld, David S., Investigation of Underlying Data: Midterm Oil and

Gas Supply Modeling System , Report No. KFR245-79, Ketron, Inc., Arlington, Virginia, January 1980.

9. Hoffman, Karla L. and Joel, Lambert S., An Annotated Restatement of the

Midterm Oil and Gas Supply System Methodology , NBSIR 80-2044, National

Bureau of Standards, Washington, D. C. , April 1980.

10.

Molineaux, J. M. , Conversion Costs: A Practical Exercise in Model Porta- bility , Kilkeary, Scott & Associates, 1980.

Note: All of the above reports are available from the Operations Research

Division, Center for Applied Mathematics, National Bureau of Standards,

Washington, D. C. 20234.
NBS-114A (rev. a- 78 )

U.S. DEPT. OF COMM.

BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO.

NBSIR 80-2128

2. Gov't. Accession No.

4. TITLE AND SUBTITLE

THE NBS ENERGY MODEL ASSESSMENT PROJECT: SUMMARY AND OVERVIEW

5. Publication Date September 1980

5. Performing Organization Code

7. AUTHOR(S)

S. I. Gass. K. L. Hoffman, R. H. F. Jackson,

Lambert S. Joel, Patsy B. Saunders

8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, DC 20234

if* Unit No.

11. Contract/Grant No.

12. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street, City, state, ZIP)

The Office of Analysis Oversight and Access

U. S. Department of Energy

Washington, DC 20461

13. Type of Report & Period Covered

i4. Sponsoring Agency Code

15. SUPPLEMENTARY NOTES

I | Document describes a computer program; SF-185, FIPS Software Summary, is attached.

16. ABSTRACT (A 200-word or leas tactual summary ot moat significant information. If document includes a significant bibliography or

literature survey, mention it here.) This report is a summary of the activities and technical reports for the Energy Model

Validation Procedure Development project undertaken by the Operations Research Division for the Department of Energy, using DOE’s Midterm Oil and Gas Supply Modeling System (MOGSM) as a test vehicle. The reports cover: (1) assessment of the documentation of MOGSM; (2) analysis of (a) ‘the model methodology, (b) characteristics of the

input and other supporting data, (c) statistical procedures undergirding construction

of the model, and (d) sensitivity of the outputs to variations in input; as well as

(3) guidelines and recommendations for the role of these in model building and developing procedures for their evaluation.

17. KEY WORDS (six to twelve entries; alphabetical order; capitalize only the first letter of the first key word unless a proper name;

separated by semicolons)

Assessment; documentation; energy; forecasting; mathematical models; sensitivity

analysis

18. AVAILABILITY [x! Unlimited

I | For Official Distribution. Do Not Release to NTIS

I | Order From Sup. of Doc., U.S. Government Printing Office, Washington, DC

20402, SD Stock No, SN003-003- DC Order From National Technical Information Service (NTIS), Springfield,

VA. 22161

19. SECURITY CLASS

(THIS REPORT)

UNCLASSIFIED

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

21. NO. OF

PRINTED PAGES

41

22. Price

$6 .00

USCOMM-OC
«

irJBH mm .