NBSIR 80-1971

Interim Report on Model

Assessment Methodology:

Documentation Assessment

Saul I. Gass

Karla L. Hoffman

Richard H. F. Jackson

Lambert S. Joel

Patsy B. Saunders

Operations Research Division

Center for Applied Mathematics

National Bureau of Standards

Washington, DC 20234

January 1980

Technical Report to: Office of Analysis Oversight and Access

Department of Energy

Washington, DC

NBSIR 80-1971

INTERIM REPORT ON MODEL

ASSESSMENT METHODOLOGY:

DOCUMENTATION ASSESSMENT

Saul I. Gass

Karia L. Hoffman

Richard H. F. Jackson

Lambert S. Joel

Patsy B. Saunders

Operations Research Division

Center for Applied Mathematics

National Bureau of Standards

Washington, DC 20234

January 1980

Technical Report to: Office of Analysis Oversight and Access

Department of Energy

Washington, DC

U.S. DEPARTMENT OF COMMERCE, Philip M. Klutznick, Secretary

Luther H. Hodges, Jr., Deputy Secretary

Jordan J. Baruch, Assistant Secretary for Science and Technology

NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director

INTERIM REPORT ON MODEL ASSESSMENT METHODOLOGY

DOCUMENTATION ASSESSMENT

Saul I. Gass

Karla L. Hoffman

Richard H. F. Jackson

Lambert S. Joel

Patsy B. Saunders

Operations Research Division

Center for Applied Mathematics

National Bureau of Standards

Washington, D. C. 20234

Technical Report to

Office of Analysis Oversight and Access

U. S. Department of Energy

ABSTRACT

This report assesses the documentation of the Department of Energy's Midterm

Oil and Gas Supply Model. The objective here is not merely to assess documentation, but also to present a method for documentation assessment. This

investigation has resulted in guidelines which can be used both to assist project sponsors in determining their documentation needs, and as a standard

against which to compare existing model reports. The documentation guidelines

presented here amplify but do not alter substantively the DOE "Interim Model

Guidelines" of December 1978 and are organized according to levels associated

with the information needs of various phases of model operation. The documentation of the Midterm Oil and Gas Supply Model is discussed in light of these

guidelines. The guidelines are recommended for incorporation into DOE model

development projects.

TABLE OF CONTENTS

1. Introduction 1

2. Phase-Structured Documentation 7

2.1 Phased Documentation Organization 8

2.1.1 Level I: Rote Operation of the Model 8

2.1.2 Level II: Model Use 9

2.1.3 Level III: Model Maintenance 9

2.1.4 Level IV: Model Assessment 10

2.2 Document-Type Descriptions 11

2.2.1 Operations Manual 11

2.2.2 Data Base Description: Physical/Logical

Characteristics 12

2.2.3 Software Description: User Level 12

2.2.4 Mathematical Description 13

2.2.5 Data Requirements Report: Sources, Transformations,

and Justifications 13

2.2.6 Process Description 13

2.2.7 Software Description: Programmer Level 14

2.2.8 Maintenance Log 14

2.2.9 Assessment Report 15

2.2.10 Model Application Report 15

2.2.11 Model Summary 16

2.2.12 Historical Record 16

2.2.13 Other Documents 17

2.3 Information Organization vs. Laundry Lists 18

3. Documentation Assessment 20

3. 1 Assessment of Level I Documentation 20
3.1.1 The Operations Manual 21

3.1.2 Data Base Description: Physical/Logical

Characteristics 22

3.1.3 Software Description: User Level 23

3.1.4 Degree of Attainment of Phase I Objectives 23

3.2 Assessment of Level II Documentation 23

3.2.1 Mathematical Description 24

3.2.2 Data Requirements Report: Sources, Transformations,

and Justifications 29

3.2.3 Process Description 31

3.2.4 Degree of Attainment of Phase II Objectives 32

3.3 Assessment of Level III Documentation 32

3.3.1 Software Description: Programmer Level 32

3.3.2 Maintenance Log 33

3.3.3 Degree of Attainment of Phase III Objectives 33

3.4 Assessment of Level IV Documentation 34

3.4.1 Assessment Report 34

3.4.2 Historical Record 36

3.4.3 Model Summary 36

3.4.4 Model Application Report 36

3.4.5 Degree of Attainment of Phase IV Objectives 38

4. Recommendations and Conclusions 39

5. Bibliography 45

Appendix A: Work Statement

Appendix B: "Approved List" Memo

Appendix C: Expanded Source Document List
1. INTRODUCTION

This first interim report from the National Bureau of Standards (MBS) project

for "Energy Model Validation Procedure Development" documents our performance

of Task 2 of the project work statement (see Appendix A), which requires the

assessment of the documentation of the Midterm Oil and Gas Supply Model. This

report records that assessment and presents a method for the assessment of

model documentation as an independent step in model assessment. Thus, this

report is an initial response to the generic concerns listed in our work

statement; viz., "to develop and apply standards and procedures for the validation of analysis systems utilized by the Energy Information Agency (EIA) of

the DOE."

We approached both our task 2 and the generic methodology-development task

from the viewpoint of model assessors. We sought answers to the following

basic questions which arise in the assessment of a model.

1) What was the model supposed to be? (Documentation accompanying the

model is the only proper source of such information.

2) What did the model turn out to be? (Computer code is a necessary but

not sufficient source for this information.)

3) Is the resulting form consistent with the intent?

4) What are the alternatives to this model?

5) What are the relative strengths and weaknesses of the model and its

alternatives?

^Sponsored by the Department of Energy, Office of Analysis Oversight and Access, Interagency Agreement No. EA 77-A-O 1-6610
-2 -

Task 2 is concerned with establishing a sufficient understanding of the code

and the conceptual model for us to answer questions 1, 2, and 3 above.

Questions 4 and 5 are clearly important but outside the central scope of task

2. They will be addressed in later reports.

Initially, our plan was to examine the documentation of the conceptual model

and its implementing computer programs, so as to understand the model and its

outputs, and to set up and execute the computer programs with several scenarios. Unfortunately, the set of documents transmitted to us by DOE (see

Appendix B) did not contain sufficient information to accomplish this plan.

We concluded that it would not be possible to complete the assessment project

without additional documentation. Therefore, to make it possible to achieve

the research objectives of the project, we expanded our source materials by

identifying and acquiring several ancillary documents; Appendix C contains a

list of the documents we obtained.

Because this material was not organized in any concise or cohesive manner, its

assimilation became a stumbling block. When augmented by information obtained

from consultations with the model developers and DOE staff, the material provided us with sufficient understanding of the conceptual model and its realization to allow the assessment project to proceed. We judged that by itself

the expanded list of documentation would have imposed inordinate effort and

time on the task of model assessment.
-3-

Our documentation assessment efforts were aided greatly by previously suggested guidelines for model and computer program documentation [1,2,5]. Thus, our

initial documentation assessment approach was simply to examine the DOE documents and review them in light of these earlier documentation guidelines. We

would have liked to have performed field-tests to determine whether documents

that conform to the suggested guidelines were sufficient for the needs of model users and of model assessors. However, the information in the DOE documents was not close enough to the information requirements of the suggested

documents for this to be done. We did complete a review of the DOE documentation and used this experience to modify the guidelines. The resultant guidelines have not been tested by their use in a model development project. Even

so, we do feel that their adoption by DOE will improve the value of documentation and the utility of DOE models.

Before proceeding with the development of recommendations concerning specific

documental requirements, we wish to comment on the importance of good documentation and on the practical question of documentation "workload." It is generally understood that the preparation of proper documentation for a largescale mathematical model is at best difficult, time consuming, and expensive.

Ironically, production of explicit descriptions of a model's minutiae, supporting analyses, and modes of operation may require a more comprehensive

grasp of the model's principles and their realization than does the model

development itself. The modeler experiments in constructing the "final" model; the documentor does not have the luxury of experimentation. Slight miscalculation in the construction of a model might escape deleterious consequences because of "compensating errors" or its submergence below the "noise
-4 -

level" of a process or data, but a flawed description of a ” delivered" model

will cause serious inconvenience or worse for users. It is not unusual for

this to be so -even when the users are the model developers! The importance of

documentation can be briefly conveyed in terms of the extreme case of virtually no documentation. Without documentation, a model is a black box to which

one does not know how to pose questions and whose responses cannot be interpreted fully. Moreover, if documentation of the computer procedures is lacking, the black box will be sealed and inert; no questions can be asked nor any

answered.

The sentiments above are seldom if ever disputed in principle, but models continue to be produced wholesale, accompanied by documentation that is scanty,

disorganized, and unclear or even misleading.

Why is this so in spite of the great amount of conscientious effort in modeling projects? Typically, modeling activity takes place with funding restrictions and under the pressure of time deadlines. Project sponsors frequently

demand ad hoc exercise of models still under formulation for "crash” investigation of special questions, or they prescribe shifts in modeling objectives in

midstream. Modelers complain with justification that they do not have time

for production of proper documentation. Moreover, it is in the nature of

problems which require complex large-scale models for attempts at solution,

that the magnitude of associated uncertainties ensures that a "perfect” or

"finished" model can never be achieved. Given this unpleasant circumstance,

modelers would always rather expend any available increment of effort toward
-5 -

refinement of the model than for documentation. One purpose of the guidelines

is to remind modelers that there is a point at which the pursuit of the will

o’ the wisp of perfection must be temporarily abandoned in favor of taking

stock and preparing signposts to allow others to follow the trail.

There are also professional considerations. In all sciences the requirement

to elaborate clearly the procedures for replication of experiments or analysis

supporting published results is not merely good practice. It is a binding obligation imposed by the standards of the academic community. Moreover, research models are customarily developed in order to buttress a particular

statement or theory, while models such as the energy models considered here

will in general be used many times by their developers as well as many others.

Thus, models produced under aegis of the federal government are first of all

likely to be subject to acceptance criteria analogous to those for various

kinds of hardware procurement, i.e. they must be accompanied by information

assuring feasibility of use and maintenance. If models are intended for use

as decision aids in support of non-trivial policy matters, they are also subject to outside critical scrutiny by "interested parties," i.e. any groups upon whom the consequences of policy decisions may have real effect. In fact,

there is a statutory requirement for public accessibility of the energy models

which are the nominal subject of this report. Although accessibility has not

yet been defined beyond quibble, there is no question that it implies open

availability of sufficient information for comprehension of the model and its

outputs
-6-

The balance of this report contains the following material: Section 2 sunma

r.zes a general methodology for documentation assessment; Section 3 presents

an evaluation of the documentation of the DOE’s Midterm Oil and Gas Supply

Model using this methodology; Section 4 presents the conclusions drawn from

our efforts to date; Section 5 is a bibliography. The appendices contain, r

spectively, the work statement for the over-all project, the memorandum containing the initial list of documentation, and a list of the documents that

were evaluated.
-7-

2. PHASE-STRUCTURED DOCUMENTATION

We believe that the currently available documentation of the DOE Midterm Oil

and Gas Supply Model does not contain sufficient information, organized in a

readily accessible format, to facilitate model assessment, he have justified

that judgment by developing and applying a formal process for documentation

assessment. To this end we have classified model operation into four phases,

each identified by its information requirements and associated documents.

We believe that the documentation required to use a model depends on the importance and complexity of problems addressed through it. For example, a

simple model intended to mechanize the analysis of a noncontroversiai system

may not require extensive documentation. In fact, in many cases, the full

amount of necessary information can exist in the form of comment cards interspersed throughout the computer code. On the other hand, a large-scale model

used for analysis supporting national policy development, and expected to undergo extensive critical appraisal, is likely to require a very complete set

of documentation. Our classification scheme is a hierarchical structure in

which the resulting set of documents for each phase or "level" includes all

the documentation in the preceding levels.
- 8-

The documentation plan presented below should be viewed as one approach to or

ganizing an acceptable set of documentation. Since many other organizations

[3,4] are possible, one need not evaluate the docunentation of a particular

modeling project by whether or not each of the designated documents exists,

but rather by whether or not all the information required in each of these re readily.* ports can be found * On the other hand, for intercomparison of a col

lection of models the benefits of a uniform standard for the organization of

model documents are indisputable.

2. 1 Phased Documentation Organization

We next describe the four phases (or levels) of documentation, listing the

document types associated with each. Detailed descriptions of the documents

are given in Section 2.2.

2.1.1 Level I: Rote Operation of the Model

Phase one is concerned with requirements for rote execution of computer runs,

i.e. the "ground rules for setting up and running the model" on a particular

computer and for verifying the correctness of the execution. The document

types are:

*Some of the designated documents could be quite brief, particularly if they

provide references to open literature sources of greater depth. Moreover,

those reports intended to be tutorial in nature could be incorporated into

their technical counterparts as introductions, if their functions are clearly

highlighted.
-9-

(1-1) Operations Manual,

(1-2) Data Base Description: Physical/Logical Characteristics, and

(1-3) Software Description: User Level.

2.1.2 Level II: Model Use

Phase two provides an explanation of a given set of scenarios and enables construction of new scenarios and interpretation of this output. The relevant

documents are those specified in I above and:

(II-l) Mathematical Description

(II-2) Data Requirements Report: Sources, Transformations, and

Justification, and

(II-3) Process Description.

2.1.3 Level III: Model Maintenance

In this phase, the documentation addresses modification* of the computer code

(and perhaps, therefore, the conceptual model) to investigate scenarios which

range beyond originally conceived limits or assumptions. The relevant documents are those specified in I and II above and:

(III-l) Software Description: Programmer Level, and

(III-2) Maintenance Log.

*We adopt hereby the assumption that a "major" change in a model's structure

ipso facto defines a new model, which would require new documentation, although this could be produced in part by modification of original documents.
2.1.4 Level IV: Model Assessment

In order to conduct a third-party assessment, model assessors should have all

model documentation available to them, especially during the documentation

assessment phase. While some of the documents listed below are not written

specifically with assessors in mind (e.g. the IV-2 is aimed at the policy

maker), they are invaluable as aids in understanding how a model was, and is

intended to be, used. However, even in the case where a sponsor determines

that a model is not to be assessed, many of these reports should still be produced. In this case, the reports would be assessed in terms of the intended

uses outlined in sections 2.2.9 through 2.2.12. The criterion for acceptable

documentation is modified here since some of the following documents may not

exist either because the work they describe has not been performed, as in the

case of (1) and (2) below, or because the potential benefits of the report to

a particular project might be judged insufficient to justify their creation,

as in the case of (3), (4), and (5). The relevant documents are those specified in I, II, and III above and:

(IV-1) Assessment Report,

(IV-2) Model Application Report,

(IV-3) Model Summary

,

(IV-4) Historical Record, and

(IV-5) all other documents written about the model and not specifically

listed above.
- 11-

Note that the hierarchy of levels goes from I (lowest) to IV (highest) but

that the order of listing of documents within a level has no intended significance.

2. 2 Document-Type Descriptions

The following descriptions of the documents listed above have evolved from

those originally presented in [1], [2], and [5] to the present form as a result of the partial field-testing performed using the documentation of the

Midterm Oil and Gas Supply Model. The documents are described here in the

order in which they were listed above.

2.2.1 Operations Manual

The purpose of this manual is to provide computer operations personnel with a

description of the software and of the operational environments so that the

software can be run. The operations manual should include an overview of the

software organization, the program and file inventory, a list of the kinds of

runs possible, a description of program control flow, a list of the run stream

control statements, estimated run time and turnaround time (at a particular

installation), an annotated list of files created or changed, and concise information on size, number, and type of output reports. For more information

on this document, see [1].
-12-

2.2.2 Data Base Description: Physical/Logical Characteristics

In [1] this document was called Data Base Specification. Its purpose is to

specify the identification, logical characteristics, and physical characteristics of the data base for the model. Highlights of the Data Base Description include special instructions for data entry, tape and file labeling conventions, support software description, logical characteristics of the data

(arrangement, relationships), and physical characteristics (storage, access).

For more information, see [1].

if 2.2.3 Software Description: User Level t

The purpose of this manual is to describe the functions performed by the

software in non-ADP terminology, so that the user organization can determine

the logistics of its applicability and how to put it into operation. It

should serve as a reference document for preparation of input data parameters

and for interpretation of results. For more on this report, see the User’s

Manual in [1].

-fr- — — - - —

Titles marked with a dagger (t) identify documents that will not ordinarily

require revision for minor modifications of a model.
-13-

2.2.4

Mathematical Description

This report describes (a) the complete details of the mathematical/logical

model, including assumptions and hypotheses, (b) the rationale for its form,

including some discussion of alternatives, and (c) restrictions on the use of

the model. This is an operational document maintained throughout the model

life cycle. Model structures are frequently modified over time and procedures

for updating this document should exist.

2.2.5 Data Requirements Report: Sources, Transformations, and Justifications

This document describes the detailed data needs of the model including input

variables and "hardwired" parameters; sources for all data, alternative data

sources, if any, and justifications for choice of data sources; processes for

obtaining data and for transforming data for model compatibility; organizational and individual responsibilities for obtaining, updating, and processing

data; numerical and forecasting techniques to be used for estimation of input

parameters; data consistency checks; and acceptable data ranges. This is an

operational document to be maintained throughout the lifespan of the model.

It should be complete in that it should give references and/or justifications

for all data; but it can reference the Mathematical Description for more detailed descriptions of the data uses.

2.2.6 Process Description!

This report should describe the background of the problem and provide at least

an outline description of the underlying physical, economic, technological,
-14-

and behavioral processes to be modeled. It includes a general description of

the problem and decision environment, provides an historical perspective, and

describes all information believed significantly relevant to the decisionmaking process regardless of whether the ultimate model includes each of these

topics. The reader should be assumed unfamiliar with the topic area and,

therefore, the report should provide definitions of terminology and references

to other source documents which more fully expand the subject areas presented.

2.2.7 Software Description: Programmer Level

The purpose of this report is to provide the maintenance programmer with the

information necessary to understand the programs, their operating environment,

and their maintenance procedures. In particular, if the model is to be used

with a computer other than the one for which the computer programs were composed originally, any specific hardware-related restrictions or characteristics should be spelled out. For more on this document, see the description of

the Program Maintenance Manual in [2].

2.2.8 Maintenance Log

This document describes the process of maintaining and updating the model. It

is a log which identifies and records changes made to the model and/or its

data, and from which can be extracted an "official” audit trail. The log

should contain the date of the change, the persons responsible for the change,

and a brief phrase identifying or naming the change. It is assumed here that

a detailed discussion of the modifications, including the reasons for and the

implications of these changes, will be included in the documentation of either
- 15-

mathematics or data. When this detailed description is available, a reference

to it should be appended to the corresponding log entry. This log should be

used throughout the life cycle of the model, beginning in the formulative

stages. It is not necessary that this log be of publishable quality, but it

is important that it be kept.

2.2.9 Assessment Report

This report includes a description of any model assessment plan agreed to by

the user/sponsor and model developer, and the results of implementing that

plan. It should include all tests of the model output in terms of comparisons

to historical data, acceptability to the user (on the basis of prior experience or intuition), statistical measures, and comparative results of alternative formulations. The developers must state and explain deficiencies and

anomalies of the model output as well as agreements with expected outcomes.

This report should at least delineate the problem environment in which the model is known to produce results acceptable to the user/sponsor, and those environments in which the results are unacceptable. The plan should be reexecuted whenever the model is changed, and this report should be updated accordingly.

2.2.10 Model Application Report

This report describes the results of exercising the model to obtain answers to

specific questions posed by decision-makers, or to study the behavior of the

problem environment as it is represented by the model. It is directed toward
-16-

the executives of the organization who will use the interpreted results of the

model to make decisions. This report should clearly and completely describe

the scenario being modeled, but need not include all of the technical details

of the model, the consequences of these assumptions, and the capabilities and

limitations of the model. Input parameters and assumptions to which the model

results are particularly sensitive should be identified explicitly. To the

extent possible, this document should quantify changes in model results associated with changes in these sensitive parameters and assumptions. Much of

this material may be summarized from the Mathematical Description, the Data

Requirements Report, and the Maintenance Log. The Model Application Report

should be of publishable quality in that it should be self-contained, it

should define terminology, and it should provide references to supporting

documents.

2.2.11 Model Summary t

This report is a nontechnical summary of the basic information that describes

the model. Its purpose is to provide, in a concise fashion, a description of

the model to a broad audience so that they may determine if it is of interest

to them. This document can be included in other documents or may be distributed separately.

2.2.12 Historical Record!

This report describes the questions or problems which led to the decision that

the model was needed, and how the model is to be used to address these issues.
-1

7-

It describes the procedures used to determine that the model can and should be

developed, including a discussion of the advantages and disadvantages of alternatives to modeling , and at least summary comparisons to alternative

models * The document should describe constraints on time, funding, personnel,

and computer facilities, that could significantly affect the nature, scope,

and approach of the modeling effort. Major participants should be identified

by name, technical background, professional affiliation, and areas of responsibility. This document is most easily prepared if it is written in the earliest stages of model conception, when the information is still current and thus

more likely to be complete. It can be constructed from memoranda, meeting

notes, and proposals, and need not be a formal publication. It should, however, be complete in that it defines terminology and references any sources of

information pertinent to this phase of model development.

2.2.13 Other Documents

It is a difficult task to determine which kind of information should be recorded for posterity and which should not. The levels-of-use concept introduced here should help in making that determination, but it is not intended to

reduce documentation decisions to a trivial exercise. In any case whichever

documents are produced should be made available to the assessment team. In

addition to the documentation set described above, there are other documents

which could be useful, and, if written, should be given to assessors. These

reports, with which we do not associate a high priority, might include software debugging plans and results, procedures for training model users, and

other reports identified in either [1] or [2] and not specifically described

herein.
- 18-

2. 3 Information Organization vs. Laundry Lists

Neither the designation of phases nor the specific organization of information

into documents should be regarded as attempts to establish definitive documentation procedures; they were selected to facilitate exposition of information requirements and because they appeared to us to give rise to a natural

grouping of model information from the viewpoint of "decision-responsible” model clients. Moreover, the list of documents should not be viewed by a model

sponsor as guaranteeing adequate documentation simply by requesting production

of "one of each of those reports." Nor, by the same token, can the developer

feel confident of providing adequate documentation just by presenting reports

containing more boilerplate than substance, which are nominally appropriate by

virtue of bearing the listed titles. Instead, model sponsors and model developers should be concerned with developing a documentation plan which insures

that model documentation provides, in a well-organized manner, all of the inf ormation determined to be appropriate for the expected level of use of a given model. These two constructs, information and organization, are the core of

proper documentation, and are of equal importance.

By information, we mean the document content which is elaborated at length

throughout this report. By organization, we do not mean merely the particular

configuration in which quantities of information are grouped or concatenated
-19-

(although considerable importance should be attached to lucid exposition within any given document). Indeed we classified information content according to

a scheme that appeared most reasonable to us, but we had relatively minor reservations against those outlined in [3] and in the interim guideline [5]. The

critically important aspect of organization relates to the ease of access to

information by a user. This means that (1) the materials in the documents

must be indexed for quick reference to any desired item, and (2) tables of

contents, and equivalently section headings, should be carefully worded to

provide a usable road map of the substance of the documentation.

We believe the lists of information requirements presented in this chapter can

be used as a checklist when evaluating model documentation, again keeping in

mind that "proper documentation provides specific and detailed information

that is organized and presented in a manner that will satisfy the needs of

each segment of a model’s audience" [2]. Furthermore, we feel that the recommended documents described and discussed in this section represent a "complete" set of documentation in that if each of these documents is produced according to accepted professional publication standards, they will provide a

prospective user institution (with a suitable computer), the capability for

independent analysis and experimentation based on the model. We will thus

have moved a long way toward developing useful models that can be used .
-20-

3. DOCUMENTATION ASSESSMENT

In this section, we evaluate the documentation (see Appendix C) of the Midterm

Oil and Gas Supply Model in terms of the guidelines developed in Section 2.

While the discussion includes the results of comparing the information presented in the reports in Appendix C to that of the ideals defined in Section

2, it should not be viewed as a report card, because the state-of-the-art of

model documentation is that standards are still under development and deficiencies in documentation appear to be universal. Rather it provides information we feel can be useful in planning future efforts. In fact, since DOE is

in the process of remedying specific documentation deficiencies, the discussion below might be used to help evaluate those documents as they are produced.

For the convenience of the reader, the discussion which follows is organized

along the same lines as the guidelines given in Section 2. In fact, the subsection numbers are the same, and we have repeated some information, viz. level definitions and document types, to avoid repeated referencing to Section

2.1. We have also included a summary of the document definitions given in

Section 2.2; these are given in italics below.

3. 1 Assessment of Level I Documentation

As stated earlier, this phase deals with the mechanics of operation of the

computer-coded version of the model on a particular computer. The relevant

document types are:
-21-

(1-1) Operations Manual,

(1-2) Data Base Description: Physical/Logical Characteristics, and

(1-3) Software Description: User Level.

Each of these will be addressed in turn below.

3.1.1 The Operations Manual

The purpose of this manual is to provide computer operations personnel with a

description of the software end of the operational environments so that the

software can be run .

The document "Operations Guide: Midterm Oil and Gas Supply Model" [C-23]

(dated 2/79) has as its stated purpose to provide "the instructions and reference information necessary to use the DOE Midterm Oil and Gas Supply Model at

a site other than the DOE/EIA Facility." This purpose was not achieved. We

have been able to complete a computer run of the model, but only after repeated sessions with DOE staff; it would not have been possible to get that far

without such close and repeated assistance. One of the major deficiencies in

this draft version (2/79) of the Guide is that the two most important Appendices, B and D, are not included in the document. Appendix B was meant to provide the information needed to read the tape files into the computer, and

Appendix D was meant to include sample outputs. We understand that the appendices were not included due to their great bulk. We feel that it is incumbent

upon authors of such a

^Lettered references can be found in Appendix C.
-22-

document to discover ways to abridge such information so that it can be included. Otherwise a document is incomplete and must remain so.

There is a disturbing assumption in the Guide that certain structures and

software systems are "standard” across IBM machines. This should be verified

(or noted to be true), since the existence of IBM's utility software IEHMOVE,

IEBPTPCH, and SORTD as fixed, unchanged routines across similar IBM equipment

is a prerequisite for this phase of model reproduction. We are also concerned

about the assumption that "the overlay options of IEWL" and the resultant

overlay structure used in the model's software are the same across IBM installations. Again, if this is true, some statement to that effect should be included. More importantly, if it is untrue, the notion of providing access to

the model via this document is an incorrect one, unless alternatives are presented.

We understand that this manual has recently been revised to include complete

listings of the Job Control Stream needed to complete a computer run, as well

as sample problems sufficient to make clear the format of both input and output and their interpretation. Statements about core storage, portability of

IBM utility packages, tape-drive requirements and other system-related features of the code are said to have been added. We have not reviewed this revised document because a copy of it was received after our cutoff date for inclusion in the documentation assessment.

3.1.2 Data Base Description: Physical/Logical Characteristics

In [1] this document was called the Data Base Specification . Its purpose is

to specify the identification, logical characteristics, and physical characteristics of a particular data base.
-23-

As of the time of this documentation assessment (2/79), there were no documents provided that satisfied these documentation needs. The "Systems Installation and Operations Guide” [C-7] contains minimal information on data set

names and sizes but no information on meaning, logical characteristics, data

entry, update methods, or references. The "Medium-Run Oil and Gas Supply Model, 1977 Data Update" [ C—1 1 ] (hereafter referred to as "1977 Data Update") is

also inadequate to this task. It contains no specific data base information

of the type described above.

3.1.3 Software Description: User Level

The purpose of this document is to describe sufficiently the functions performed by the software so that the user organization can determine the logistics of its applicability and how to put it into operation. It should serve

as a reference document for preparation of input data and parameters and for

interpretation of results .

There was nothing in the materials furnished to us that approximated this

document.

3.1.4 Degree of Attainment of Phase I Objectives

The delivered document materials do not satisfy any phase I requirements.

3. 2 Assessment of Level II Documentation

The second level identified in Section 2 is the one in which one tries to use

the code to implement the model. In this situation, one needs sufficient
-24-

information so as to be able to run the model with new input data. The relevant documents are those specified in Level I above and:

( I I— 1 ) Mathematical Formulation Description;

( I 1—2 ) Data Requirements Report: Sources, Transformations and

Justification; and

( I 1—3 ) Process Description.

The Midterm Oil and Gas Supply Model reports will now be evaluated in light of

these document types.

3.2.1 Mathematical Description

This report describes the complete details of the mathematical/logical model

including assumptions and hypotheses, the rationale for its form, and the re- strictions on its use.

There is no of documentation for the Midterm Oil and Gas Supply Model that

serves the function of the Mathematical Description. Some examples of the

omissions in the existing documentation will be presented below, in order to

illustrate the need for a more complete and better-organized approach to the

documentation of this model. Each item in the definition of the ideal above

is addressed separately below.

3.2. 1.1 Mathematical/Logical Structure of the Model, Including Assumptions

and Hypotheses

There are four separate documents [C-10, 11, 12, 22] which present pieces of

the mathematical/logical structure of the Midterm Oil and Gas Supply Model, as
-25-

well as a number of ancillary documents [C-14, 15, 16, 17, 18, 19, 20, 21]

which contain some more detailed descriptions. In order to understand the

model’s representation of the underlying process of oil and gas supply, all of

these documents must be read. Even though approximately twenty documents describing the Midterm Oil and Gas Supply Model have been studied, some calculations performed in the model remain unexplained. For example, the "Oil and

Gas Midterm Supply Model: Methodology Description" [ C—1 2 ] (hereafter referred

to as the "Methodology Document") describes a number of cash flow categories

such as investments (expensed and capitalized), production costs and annual

cash expenses (depreciation, depletion) that must be used in determining the

cost of producing various quantities of oil and gas. Knowing how these data

items are used is crucial to an understanding of the model, but this usage is

unexplained.

One report, "U. S. Oil and Gas Supply Computer Program Documentation" [ C—2 2 ]

,

dated 1973, has been cited to us as the documentation of the mathematical

structure of the model. The only legible section of this document, Section

VI, was added at some later time by staff of ICF, Inc., the contractor who

prepared the model, to detail the changes made to the original model.

Unfortunately, that section does not explain which pieces of the original

model are replaced or altered.

Section VI also presents a listing of equations and brief descriptions of

variables. However, reading a lengthy list of equations, without explanation

or justification, is not much easier than reading computer code, nor more

enlightening. For our assessment efforts, we need additional elaboration of
-2 6-

the reasoning behind the mathematics of each section of the model. A useful

guideline could be the comments related to equations 98-107, found in Section

VI (page 5.20) of that report.

A description of the model should not only include the mathematical statement

of the model but should discuss each of the assumptions made. This discussion

should include for each assumption, an explanation of its effect on model output values. In the case of the Midterm Oil and Gas Supply Model, assumptions

are scattered throughout various documents. They are nowhere collected or

classified, and no indication is given as to how they individually and jointly

limit the applicability of the model. The model can and should be used to

study the effects or implications of some assumptions, and the results of such

a study should be reported to policy makers.

This lack in the model documentation is especially serious in a policy making

environment where the decision makers are not the analysts. Without information about the assumptions of the model and any competing models, it is impossible for a decision maker to choose among alternative models or even to

have confidence in the output of a specific model.

Although many assumptions are articulated in the Midterm Oil and Gas Supply

Model documentation, others are left unstated or their implications are not

explained. For example, decline rates are assumed to be constant. The implication of a constant decline rate is that regardless of future price
-2 7-

increases, there will be no extraordinarily intensified extraction from old

fields which could affect the quantity of oil produced. This potential effect

should be discussed further in the report.

We recommend that a list of assumptions be compiled and checked for completeness by the model analysts and computer programmers responsible for the final

model and computer code.

3. 2. 1.2 Theoretical and Analytic Rationale for the Mathematical/Logical Form

Including Some Discussion of Alternatives

None of the documents discuss alternative approaches, nor are discussions presented for any of the equations used in the model. We present one example to

illustrate why we believe each of the equations should be discussed in more

detail.

Hubbert's factors are used to calculate growth of the resource base from "extensions and revisions," but no justification is presented as to why these

factors are appropriate or why national figures can be applied to all regions.

Sensitivity of the output to these factors is not reported. Furthermore, no

reference to the original work of Hubbert is provided, thereby making the uninitiated assessor’s job of evaluating these formulas much more difficult.

Similar criticisms apply for each equation used in the model.
-2 8-

In addition to providing the rationale for the form of each piece of the model, some documentation describing the choice of a general engineering-process

approach as opposed, for instance, to an econometric approach should also be

presented. This discussion should include brief narratives that describe in

general terms the alternative models available and the strengths and limitations of each. This should not be a relatively expensive and time-consuming

effort. Any good research or development project should begin with a literature search to discover what has already been done. This should be summarized

for the record.

3.2. 1.3 Restrictions on the Use of the Model

The documentation of the Oil and Gas Model contains no explicit statement of

the time horizon for forecasting. We consider this a serious omission. The

abstract of the Methodology Document states "the Oil and Gas Supply Model is a

computer-based model which projects domestic oil and natural gas production

for 15 years based on economic and engineering factors which affect oil and

gas supply." One question which naturally arises is whether this model can be

used to predict gas supply in either the short term (two or three years from

the present), or the longer term (twenty-five to thirty years), or if the underlying methodology requires that model outputs be considered valid only

after some adjustment period. Statements describing modes of applications of

this model and its limitations would be useful.

Since the Model Formulation Description is intended to serve as the primary

source of information about a model, we recommend the preparation for the Midterm Oil and Gas Supply Model of such a document along the lines indicated
-29-

above. This document should include a description of each of the items listed

in the ideal above and should include references to any other documents needed

to understand, use, validate, or alter the model.

3.2.2 Data Requirements Report: Sources, Transformations and Justification

For all input variables and "hardivired" parameters, this report describes data

sources, transformations and justifications thereof, data validation procedures and acceptable data ranges .

The 1977 Data Update [C-ll] supplies some of the information described above,

but omits some data values and is incomplete in its discussion of others. For

example, according to the Methodology Document, in order to calculate the segment of the resource base which is economic to pursue, "a special run of the

engineering and minimum acceptable price sectors of the model are executed

with a hypothetical, ambitious drilling program." No definition or explanation of a "hypothetical, ambitious drilling program" is provided, nor is the

purpose of executing this run explained. Similarly, although there is a

lengthy discussion in the Data Update on how to allocate drilling among old

and new rigs, no data is provided for the inital rig and plant capacity.

The 1977 Data Update does not give sources for all of the data; e.g., the offshore escalation factor is given as 2 percent, with no justification or reference. While the report includes a detailed, yet incomplete, discussion of the

processes for transforming data into the finding rates needed by the model, it

is less specific in describing other necessary transformations. For example,
-30-

calculation of the secondary and tertiary recovery and intensification factors

involves some unexplained application of Hubbert factors, utilizes referenced

but undefined tertiary recovery production forecasts, and applies some unspecified adjustment factors.

Numerical, statistical, and forecasting techniques used for parameter estimation are not described. For example, in describing the calculation of finding

rates the report states that regression analyses were used and includes tables

of the results of these analyses. The document does not specify, however, the

method of regression, the regression code used, or the measures of significance associated with the parameter estimates. Finally, there is no description of any data validation efforts nor is there any attempt to identify

reasonable ranges on the data. Although the report does characterize by

"high, medium, or low" the sensitivity of the model to the data items on

several lists, this information is of little value for assessment purposes

without an accompanying description of how these sensitivities were determined, including numerical ranges for the data items, identification of which

output items were sensitive to changes, and quantitative definitions for the

measures of sensitivity.

A data requirements description is an essential document, and it is strongly

recommended that DOE initiate efforts to produce one. The 1977 Data Update

would supply a framework and some of the necessary information, but it needs

much improvement and polishing before it will satisfy the data information

needs of users and assessors.
-31-

3.2.3 Process Description

This report provides a comprehensive description of the underlying physical,

economic, technological , and behavioral processes to be modeled.

In the case of the Midterm Oil and Gas Supply Model, the purpose of the process description is to provide an introductory, but thorough, description of

engineering, economic, and political factors that are believed to affect the

future national supply of oil and gas.

There is no such document for the model and although the Methodology Document

includes a section on the supply process, it does not describe adequately all

aspects of the process, nor identify an existing source for this information.

Central topics omitted from the discussion in the Methodology Document include: the effects of regulation and taxation on oil and gas supply; the

present configuration of, and the barriers to entry into, the industry; the

differences between the geological estimates and resource base; and the manner

in which mineral property rights are acquired through leasing and royalty

agreements.

In contrast, three reports "A Comparative State-of-the-Art of Assessment of

Gas Supply Modeling" [C-3], "A Comparative State-of-the-Art of Assessment of

Oil Supply Modeling” [C—4 ] , and "Oil and Gas Resources—Welcome to Uncertainty" [C-23] provide a more complete description of the underlying process being

modeled. If these reports accurately portray the oil and gas supply process,

they should be referenced as sources for such process information. If not,

DOE should consider producing such a report, comparable in depth but rectifying existing deficiencies
-32-

3. 2. A Degree of Attainment of Phase II Objectives

The delivered documentation does not furnish any capability for experimentation beyond rote (Phase 1) operation.

3.3 Assessment of Level III Documentation

The third level of model use identified in Section 2 is the model maintenance

phase. During this phase one attempts to make changes to the existing model.

The two suggested reports are:

(III-l) Software Description: Programmer Level; and

(III-2) Maintenance Log.

These will now be discussed.

3.3.1 Software Description: Programmer Level

The purpose of this document is to provide the maintenance programmer with the

information necessary to understand the code architecture, the logic of each

of the subprograms, and the operating environment to a sufficient depth so that the programmer is capable of maintaining, correcting, and enhancing the

computer code.

This report is an important information source to anyone concerned with assessment, as well as to those who must operate and run the model. Whereas the

"Software Description: User Level” (see sect. 2.2.3) provides an overview and

instructions on how to use the computer realization of the model, the programmer level manual gives details of the programs themselves.
-33-

The report "U. S. Oil and Gas Supply Computer Program Documentation" [ C— 1 4

]

can serve as an example but, unfortunately, not as an operational embodiment

of the programmer-level document. It is dated 1973 and contains no reference

to subsequent updates. There is no obvious one-to-one correspondence between

the subroutines, data input, and output reports described in the document and

those of the computer software we received. Furthermore, the available copies

of this report are largely illegible.

3.3.2 Maintenance Log

This log identifies and records changes made to the model and/or its data .

For the assessment team, this log would be most useful in that it would provide a chronological description of model evolution. Although it is most difficult to make log entries during a crisis, procedures exist according to

which it is done routinely, e.g., by air traffic controllers in saturation

traffic, and for various analogous activities. Log maintenance is especially

important in "crash mode," that is, in periods of tense activity and rapid

changes (when working against deadlines) to guard against chaos "when the

shooting dies down." There is no documentation of this type in any of the

reports listed in Appendix C.

3.3.3 Degree of Attainment of Phase III Objectives

The documentation provides no assistance for modification of the model.
-34-

3.4 Assessment of Level IV Documentation

This level of documentation contains information needed for conducting a

third-party (or arm's length) model assessment. The documents included in

this section address topics relevant to the assessment effort such as reports

of efforts to apply, verify, or assess the model. It is also the catch-all

for any other reports relating to the model, since all model documentation

should be reviewed by the assessor. Typical reports to be found in this broad

category include:

(1) Assessment Report,*

(2) Historical Record,

(3) Model Suomary,

(4) Model Application Report, and

(5) any of the documents in [1,2,5] or any other documents written about

the model but not specifically listed above.

The first four documents in this category are discussed below.

3.4.1 Assessment Report

Documentation of model assessment activities should include the results of any

tests of the model's output in terms of comparisons to historical data, ac- ceptability by the user, statistical measures, comparative results of alternative formulations .

*Note that this document refers to assessment in which the model developer

participates, at least in the planning stage.
-3 5-

In the case of the Midterm Oil and Gas Supply Model, documentation in this

area is apparently nonexistent. Two prof erred documents, however—"FEA Model

of Oil and Gas Supply: Data Validation and Update " [C-14], and the 1977 Data

Update—list three levels of sensitivity (high, medium, low) of model outputs

to variations in values of selected input parameters. If these were arrived

at judgmentally , the documentation should so state. If these estimates resulted from some testing, the tests should be described. Similarly, acceptance testing of the computer code and statistical tests of the regressions

may have been performed, but documentation describing these test efforts has

not been provided. We recommend documentation of all validation procedures

performed on this model and of the test results which can be reconstructed at

this time.

Obviously, if proper testing is performed at appropriate stages of a model's

life cycle and if this testing is well-documented, an assessor's job is

easier, and the assessment is more likely to be cogent and complete. Without

such documentation, the assessment team must recheck much that may already

have been checked. Extending this notion, if the model developers performed

all the validation and verification tests possible, and provided documentation

of those efforts along with the other documents requested in this report, assessment efforts could be limited to checking for reproducibility.
-3 6-

3.4.2

Historical Record

This report describes the questions or problems which led to the decision that

the model was needed, how the model is to be used to address these issues, and

the advantages and disadvantages of alternatives.

None of the documents listed in Appendix C contain historical information of

the type described herein for the DOE Midterm Oil and Gas Supply Model. Absent documented input from the policy makers, this information is best known

to3.4.3the model builders and should, therefore, be constructed by them.

Model Summary

This report provides a concise summary of the model so that other users and

analysts can determine if it is of interest to them.

The document entitled "Description of Method Used to Forecast Domestic Oil and

Gas Supply” [C-4] satisfies this need. It is a concise overview of the model

and should become part of the formal documentation. We recommend that a reference be provided to the work of Hubbert and that definitions be provided in

this summary for the following technical terms: resource base, new pays, inferred and indicated resources, and recovery methods.

3.4.4

Model Application Report

This report, directed toward those who would use the model results to make decisions, should present results of exercising the model to obtain answers to

specific questions posed by executives and to study the behavior of the problem environment as it is represented by the model.
-3 7-

The "Annual Report to Congress," [C-l ] is similar to the Model Application Report described above. This document, however, addresses the entire set of

models including PIES and the various models which supply input to PIES and/-

or process output from PIES. It is therefore diffcult to locate and identify

within the report those statements which apply exclusively to the Midterm Oil

and Gas Supply Model. The sections entitled "Summary," "Introduction," and

"Energy System Projections," and the "Oil” and "Gas" chapters of "The Sources

of Energy" all contain statements related to oil and gas supply, but it is not

possible to isolate from the context the role of MOGSM in the generations of

these forecasts.

In describing data values, the Annual Report fails to distinguish among model

inputs (and assumptions), model outputs, and external reference data (and assumptions). For example, that report states that domestic production of petroleum liquids is forecast to decline by about 9 percent from 1977 to 1985

(page 138) and indicates that the accuracy associated with this estimation is

"still quite high; confidence in the forecast estimate stems primarily from

the accuracy associated with the portions of the resource base that contribute

most of the reserve additions, primarily old fields." However, what is not

stated is that the decline is therefore the result of an assumption—namely,

that decline rates are constant and that the decline rate is an input value

which DOE has chosen to be the current rate of production used by the oil industry.

*Since the start of this study, the name has been changed from PIES to MEMM or Midrange Energy Market Model.
-3 8-

Because the Annual Report discusses assumptions throughout its text, and collects some but not all of these into paragraphs labeled "Assumptions," it

would be difficult to determine if all model assumptions are omitted. For example, in the 1977 Data Update, the rate of return on investment is defined in

a single short sentence as 8 percent, with no source or justification and no

ranges on this value, which apparently is taken to remain constant over all

years, regions, recovery methods, etc. That same report identifies the rate

of return as an input parameter to which model sensitivity is "Hi." Yet

neither this sensitivity, nor the particular value (8 percent), is mentioned

in the previously referenced sections of the Annual Report.

3.4.5 Degree of Attainment of Phase IV Objectives

While "model assessment" is not at present completely defined, it is clear

that signficant components of an assessment process—operation of the computer

programs, comparison of the rationale of the model's structure and model outputs to the model objectives, determination of limitations in range and

scope —cannot be accomplished by using the computer programs and the documentation alone. Direct assistance by the developers is necessary.
-39-

4. RECOMMENDATIONS AND CONCLUSIONS

Our assessment of the documentation of the Midterm Oil and Gas Supply model

has led to several conclusions and general recommendations. First, on the

subject of documentation assessment, existing guidelines for evaluation of

models and for evaluation of supporting documentation do not constitute a

clear application of a highly developed theory; they are merely common sense

clothed in formal language. In this absence of a definitive set of rules and

criteria, we recommend that others undertaking a similar effort follow our approach. That is, we decided on a general description of the documentation

necessary for assessment, and then compared the available documents to this

description. This recourse to a standard, even one possibly subject to eventual drastic revision, is the first step in transforming the assessment of

model documentation from the realm of literary criticism to that of a measurement process. Second, also concerning assessment, we strongly recommend that

at least one member of the team assessing documentation should be unfamiliar

with the model, thereby relying on the documents as the only source of information about it. Persons already familiar with the model may take for granted

some piece of information and thus fail to recognize its omission from the

documentation or any lack of clarity in the exposition.

On the subject of model assessment: after considerable effort, we have been

able to obtain an overview of the Midterm Oil and Gas Supply Model, sufficient

for a partial model assessment . This is not to say that the documentation is

sufficient. Indeed, much labor and information acquired outside the documents

were required to reach our current state of model understanding, beset with

gaps as it still is. A fuller understanding of the model would require
-40-

reading every line of computer code and having the modelers explain any discrepancies between the mathematical formulation and computer implementation.

Unless better documentation is developed, others desiring to use or to assess

the model will have to repeat our efforts only to reach the same level of incomplete understanding. In essence, this means that with much effort even a

poorly documented model can be assessed if (1) those aspects of the model for

which additional information is essential can be identified and (2) this additional information can be obtained directly from persons intimately familiar

with the model (e.g. , DOE staff or contractor).

However, since we do not intend by our own efforts to add to the body of documentation of the Midterm Oil and Gas Supply Model, we must address the problem

of recommending ways to improve access to it through the poet hoc preparation

of documentation. This situation is different from advancing methods and

techniques for the a priori development of a documentation plan for use in a

modeling project. Now, we must accept that for some time to come, models (for

example this model) will be developed, delivered, and used with documentation

that does not satisfy any of the guidelines under development. In Section 3,

we discussed each recommended document type and evaluated this model's documentation in light of that set. We also indicated documentation which seems

to be crucial to the process of model assessment, and should therefore be prepared after the fact if it has not been done during model development. This

"post hoc" set is collected below:
-41-

For Phase I usage

(1) Operations Manual,

(2) Data Base Description: Physical/Logical Characteristics,

(3) Software Description: User Level.

For Phase II usage by "sophisticated" users

(4) Data Requirements Report.

For Phase III usage

(5) Software Description: Programmer Level

(6) Mathematical Description,

(7) Maintenance Log (to the extent that the pertinent information can be assembled now)

To satisfy the needs of a broad spectrum of users the Phase II post hoc document should be augmented by (8) Process Description. This document is needed

in any case for model assessment (Phase IV).

We have stated earlier the crucial importance of good documentation in providing the means for using models and the professional requirement for

enabling replication of the results of analysis. There are other benefits

to be gained from a thorough documentation effort, including:

(a) training users inside the sponsoring organization

(b) introducing the model readily to interested parties, outside the

sponsoring organization, and

(c) discovery of model enhancements when modelers structure their

knowledge of the model in order to communicate it to others.
-42-

Proper documentation cannot be taken for granted. In the introduction, we

have noted reasons related to deadline pressure and reluctance to "freeze"

models that account for scanting of documentation by model developers. This

situation is likely to endure indefinitely because of the critical nature of

the problems addressed through large scale models and the expected deficiencies in background data. Therefore, model sponsors should take actions designed to assure adequate documentation. These include at least, specification of information requirements in negotiating contracts for model development and provision of funding and time schedules for documentation as line

items in such contracts. Also, in contract awards, the same kind of scanting

of qualifications and specification of level of participation should be given

to proposed documentors as is now customarily given to analysts.

We believe that the guidelines described in this report should be provisionally adopted for modeling activity under DOE aegis because the information

content described represents substantial consensus among analysts and is not

likely to be revised in a way that invalidates documentation thus produced.

Moreover, we assert that information designated in the guidelines is essential

in that no revisions could result in substantial excess effort having been expended.

In invoking consensus, we remark that although we relied on [5] and [2] as

background, noting that the intention in [2] was exhaustive documentation for

planning, usage, evaluation, and archive purposes, we proceeded independently

to develop these guidelines from first principles, as it were (through interviews, reading, and discussion to identify information requirements for a

spectrum of application and then to assemble them according to a rational

plan).
-43-

The result differs from the material in [2] primarily in defining a more concise set of documents and having discarded a set of criteria which we regarded

as too highly precise and structured at the current stage of elaboration of

documenting methodology. The guidelines are substantially similar to those of

[5] except that they expand the document set (but not the total information

requirements) according to specific levels of model usage. A statement that

we believe that the guidelines should be adopted but that they are subject to

some change and that they should not be rigidly applied, requires some clarification. We believe that (at this time, at any rate) model documentation

giudelines cannot be stated with the force of, say, specifications for military equipment. We intend that the guidelines should become a rational basisfor a priori agreements between sponsors and developers as to the documentation that shall accompany models (and according to which mutually agreeable

criteria for judging that documentation can be derived).

Finally, we offer two conclusions of a methodological nature. We believe that

production of documentation pari passu with the development of a model will

maximize efficiency and quality of documentation even for those portions of

the material which would require modification as the model is refined. Possible omission of relevent material will be minimized, milestones will be more

easily scheduled, and no time losses should be incurred in reworking material-

-it is as unlikely that a narrative will be "right" without revision as that a

computer program can be written linearly without bugs.
-44-

As in the case of document asessment, some member of the document producing

team, preferably an editor, should be unfamiliar with the details of the model

or its methodology, and therefore be in a good circumstance to detect departures from clarity in the exposition.

A belief that has guided our efforts in this project is that if we can get

some notion of what constitutes "good" (minimal, maximal, or whatever) documentation and if this can be cast into a set of guidelines in such a way as to

avoid bureaucratic rigidity in their application and use, we have moved closer

to that goal mentioned earlier of learning how to develop useful mathematical

models that can be used.
-4 5-

5. BIBLIOGRAPHY

1. "Guidelines for Documentation of Computer Programs and Automated Data

Systems: FIPS PUB 38, National Bureau of Standards, Washington, DC,

February 15, 1978.

2. Gass, Saul I., "Computer Model Documentation: A Review and an Approach,"

NBS Special Publication 500-39, National Bureau of Standards,

Washington, DC, February 1979.

3. House, P. , and McCloud, J., Large-Scale Models for Policy Evaluation , John Wiley and Sons, New York, New York, 1977.

4. "Computer Program Documentation Guidelines," NHB-2411-1, NASA, Washington,

DC, July 1971.

5. Lady, George M. , "Interim Model Documentation Standards," Memo to Applied

Analysis Senior Staff, EIA, December 4, 19 78.

APPENDIX A: WORK STATEMENT

Energy Model Validation Procedure Development

SUMMARY

This action is an interagency agreement which initiates a

continuing activity at the National -Bureau of Standards

(NBS) intended, to support Energy Information Administration

(SXA) in model validation activities. The N3S is tasked to

make findings as to the minimum model operating and conceptual

documentation required to conduct the model validation activities,

to examine model attributes and to develop validation standards

designed to measure the ’'confidence” in model results. For this

initial effort the mid-term oil and gas supply model will be

the system considered. Particular model attributes to be

evaluated include.:: - quality o: data. — rational for the model's logical, mathematical,

and statistical properties,

• — methods, for comparing model results to known'

outcomes, and . * • — other characteristics important to developing

measures -of model confidence.

Xn-so-far as possible', rigorous concepts of model confidence

will be developed and related to the model evaluation process

.

Similar efforts, emphasizing different EIA models, are planned

for the Los Alamos Scientific Laboratory (electricity) .and the

MIT Energy Laboratory (coal)

PROJECT STATEMENT OF WORK

Title: Energy Model Validation Procedure Development

In br

o

y

UCuron = This project is part of the Office of Analysis Overaignu ar4 Access' program to develop generic standards

anc procedures foi: the assessment and validation of Energy Information Administration (ElA) analysis systems. The EIA na^ esmbmsn— many large scale mathematical and statistical procedures ~ or Pro3 acting and analyzing energy production, consumption* prices, and associated impacts. The systems are

COra?

i”x mathematical, technological, and statistical t-chn-gowo and^ara lo be subjected to extensive, continuing

re '/iswo and^ crz.-p.ques aimed at determining and improving

. tneir validity, acpiracy, and abilities. These systems will, always be m a s*_aue of chancre. It is imnn-r-r^nt -Forof each system to determine its current validity, .and to suggest improvements and state-of-the-art extensions.

version

??? ^

isns3 have the National Bureau of Standards (NBS) establish a^ continuing activity that will be tasked to develop and apply s Lancards _ and procedures for the validation of analysis systems utilized by the EIA of the DOE. The goal of this project is to develop methods for finding the degree of confidence in results and the circumstances under .which the * systems may be used. The procedural approach will be a systematic-. evaluation of a system’s assumptions, structure,

i

'v*a »_ion , alternatives, performance, sensitivity, and any Ouner factors which are significant influences upon the confidence in system results. A major task will be the determination, to the extent possible, of rigorous procedures ion ufilming such evaluations in a determination of confidence in system results in specific uses. For the- first year's effort

s oil and .gas supply foretasting systems will be used m the development .of validation procedures- After the

. successful evaluation of this system additional systems will - be chosen and subjected to the evaluations described in Tasks. 1-3 in the scope of work.

The specific objectives of this project are (1) to develop. methods,

Userut for validating EIA analysis systems; - (2) to 'establish

a team of. analysts consisting of NBS personnel and outside

consultants that will accumulate, and maintain expertise in

validation procedures 'as applied to DOE analysis systems; (3)

initially, to evaluate the DOE mid-term oil and gas supply

models; (4) to report to DOB on the results of the oil. and gas

supply model evaluation and their implications for users of

the models; and .(5) to specify systems validation standards

and procedures based on the experiences of the mid-term oil and

gas supply model- evaluation. 'After the mid-term oil and. gas

model has been evaluated additional svs tarns will be chosen by

" MM
2

The major activities of this project will be the development

of system validation standards and procedures and their

noolication. to the latest version of the mid-term oil and cja< sioplv analysis systems, ' For the purposes of this project,

existing documentation reports will be the initial input for

the evaluation process. It is assumed that the systems and

their operating computer models underwent, during and after

their initial developments, some verification, and validatio:

tests- -The- results of any such tests, if available, will be

valuable in x structuring the evaluation to be accomplished.

As it ’is not certain that previous tests have been documents

this project will establish both verification of the program

and techniques of the operating systems- Thus, the proposed

evaluation - will require the project team to have the means -t( subject the systems to detailed scrutiny, including program

listings and. the running of the computer programs under ' various conditions.

SCOPE OF V70RK

The following tasks will be undertaken by the NBS to specify

and apply the validation procedures.

Task I s.-' Existing documentation of the oil and. gas analysis

systems will be^ examined 'and project personnel will esfcablis

operating versions, of the systems for project use'.

Task 2 ^

' Operating and conceptual documentation will be

evaluated and deficiencies identified. For the purposes of

this project a documentation deficiency refers to any and

all aspects of model documentation- which is not available,

but necessary to' perform the other tasks of this project.

To the extent to which documentation deficiencies exist, sue? remedies as necessary to support this validation project wil

be undertaken- ‘ It is recognized that the extent of document

deficiencies, if -any, is not now known. As a result, the

remaining tasks of -this project are contingent upon the

successful completion ,of this task.. The resources allocated

to this cask in the performance of the project,, including the project schedule 'for this task, may differ from those

specified here due to the actual extent of documentation

deficiencies. / .
3

3 z Systems attributes will be evaluated to include:

Tar.i 3,1 Completeness and accuracy of underlying data,

Thi: sue fa sk calls lor a z unciing or -che

sufficiency cf the underlying data based u;

existing' documentation of the data? it does

not call tor an independent audit of any of the

data at issue, •

Task 3,2 Conceptual sufficiency 'of system specification.

This subtask calls for a finding as to the

completeness of the set of concepts or variables

included in the system and the completeness of

the set of interrelationships among the variables

accounted for by the model. Particular emphasis

is placed .upon the identification of alternative . specifications and the rationale for the particular

specification chosen.

Task 3.3

Appropriateness of operating- representation.

This subtask, calls for a finding as to the adequacy

of the particular mathematical forms adopted for the

model. Particular emphasis is placed on.;' the

functional or algorithmic forms employed- for

determining variable values, alternatives to -such

forms and the rationale for the particular forms

- chosen as well as those rejected- . . Task 3.4 Appropriateness of 'embodied estimation methodologies

.

‘ This subtask calls for a finding as to the adequacy

- of the- statistical or other procedures utilized to

derive the' parameter values embodied in the model’s

mathematical representation- Particular emphasis

is placed "upon alternative estimation procedures

and the rationale for the procedures selected for . the model.

Task 3.5 System- sensitivity ’and ‘stability. For each of the

area3 of model attribute identified under spacrficatior

representation, and estimation this subtask calls for

a determination of the sensitivity or other quality,

of model result associated with the particular

choices which make up the model itself compared to

the • alternative choices not made. Particular emphasis

is placed upon, a finding of the strengths and weax—

*nesse3 of the choices made compared to therr

alternatives.
4

r Pr'sk 3.6 System performance compared to known outcomes.

This subtask' calls for the identification oi How

modeling results can be verified by comparison

to known outcomes and how the results of that

comparison can be utilized in preparing measurements or other indications of confidence in - model

results- If possible r such comparisons will be

attempted- At a minimum a methodology for making

and procedure for using such comparisons will be

' developed,

Task 3-7 •. Computer related system 'characteristics -

Task" 3. & Any other system element or 'attribute which

• significantly influences the confidence 'in

. system results. .

Task 4: The results of the evaluation, will be consolidated ^ and a report on the system strengths and weaknesses prepared!*15

Task 5 - A specification of alternative concepts of

" confidence” in' system results will be prepared.

Task 6: A determination will be made of the relationship.

between the outcome of the various system attribute B evaluations and the concepts of confidence. To the extent

possible a rigorous statement of this relationship will be

achieved- • :

Task 7 z A summary concept of system result confidence will

be developed to' include the specification of -the ‘evaluation

activities necessary to support 'the determination of system

result, confidence. • •

Task 8 : An end of year report will be prepared on standards)

and procedures for determining system confidence.

After the successful evaluation of the mid-term oil 'and- gas

supply model additional systems will be chosen by- DOE and

the work recommenced "'at Task 1. '

Schedule (for first calendar year)

x asK. I-.-*. - 6 weeks after start

.xask 2 - - - ~ . 12 weeks after start

Tasks 3 and 4 . .... 8 months after start

Tasks S-fi-7. an* « -V> —U.
5

Reports for first; calendar year

TjLj titer progress repo__ s. . Interim Report on Tasks 1 and 2 ...... 3 months after start

Draft Interim Report on

rp£j 3 and .....

% ...... S months af ter s tart

Interim Report- on Tasks 3 and A 9 months after start

Draft End of Year Report. ----- 11 months after start

End of Year Report- - - - months after start

APPENDIX B: "APPROVED LIST" MEMO

/ • «— ^ / .i' \y

O. — >- '^ris o* *

UPJ JTED STATES DEPARTMENT OP CDMP/JERC

National Bureau of Standards

Washington. n.C. 20234

October l 6 , 1978

MEMORANBUM FOR George Lady

Office of Analytic Methods

Department of Energy

sn >- From: Richard H. F. Jackson

Operations Researcn Division

National Bureau of Standards

Subject: List of Documents Obtained .by BBS to Date

?

n

idp

t^Lt0 aSf“? tha* has received all pertinent documents

°, have for

. ths documentation evaluation phase of our effort, T»e provide belov the list of documents received to d~te Tt ->=? “very^t "^

1^ ****”*' m ° ^

others aie very draft. The list is in no particular order.

(1)

Energy Information Administration, Annual Report to Corigress,

T

~ _ I

’ - , r

?nfo 10ns

. 01 energy- Supply and Demand and Their Impacts, April I978 .

(2)

randS rT n

&S S

-UPPly Model > 1977 uPdate > Research Memo- ranaum No. 78-015, December 1977

.

(3)

J*

lto6?“d Used to Forecast Domestic Oil and Gas Sunplv, Unpuba-snea ffo.es Received from C. Everett, Undated.'

* "

Undated!

03 0u:t;Line, 0btainad from C. E. Everett, Untitled and

^ Gd

“,

al

.ld Supply Curves for the Administrator's Annual Report Technical Memorandum TM/ES/78-I7 , September 1978 .

’

( 6

)

Midlevel Documentation, ICF Inc. Contract Report, July 1976. (7)

Petrol eu^t

^

G

-, Petroleum

S“PPly Co:nI)uter Program Documentation, National Council, November 1973 .

(8)

'

October W,°m

^* *“ °f °U “* GaS Model * Received

(9)

Li t. Oi D..ta Inputs to Oil and Gas Model, Received October 10, 1973.
( 10 ) Unpublished Notes on Oil and Gas Model Overlap

October 10, 1973.

Structure, Received

(ll) Oil and Gas Mid-term Supply Model; Methodology Description, Draft

Technical Memorandum TM/ES/73- •

(12) Listings of the Subroutines that Comprise the Oil and Gas Model, . Received October 10, 1978, dated January 12, 1978.

Since 1IBS is required to produce a finding on the completeness of the documentation of the Oil and Gas Supply Model, ve request that any omissions from

this list be brought to our attention as soon as possible.

cc: C. E. Everett

S. I. C-ass

D. Hulett

IT. Mann

F. Murphy . . R. P. O’Neill
APPENDIX C: EXPANDED SOURCE DOCUMENT LIST

1. Annual Report to Congress , DOE/EIA 0036/2, Vol. II, 1977, United

States Department of Energy, Washington, D. C. 20461.

2. Appendix E of National Energy Outlook , titled Oil and Gas Supply,

draft copy, undated. Dept, of Energy, Washington, D. C. 20461.

3. Celiano, R. , Limaye, D. R., and Hu, D. S., A Comparative Stateof-the-Art Assessment of Gas Supply Modeling , EPRI EA-201, prepared by Mathtech, Inc., for Electric Power Research Institute,

3412 Hillview Ave. , Palo Alto, CA 94304, February 1977.

4. Celiano, R. , Fallah, M. H., and Limaye, D. R., "A Comparative

State-of-the-Art Assessment of Oil Supply Modeling," EPRI EA-1609,

prepared by Mathtech, Inc., for Electric Power Research Institute,

3412 Hillview Ave., Palo Alto, CA 94304, June 1978.

5. Everett, C. G., "Description of Method Used to Forecast Domestic

Oil and Gas Supply," undated note.

6. Everett, C. G., "Discussion Outline," undated viewgraphs of Oil and

Gas Supply Model.

7. Everett, C. G., "Model Documentation Summaries," memorandum for

George Lady, dated September 28, 1978.

8. Everett, C. G., and Mahn, N., "System Installation and Operations

Guide Midterm Oil and Gas Supply Model," draft copy of Technical

Memorandum, dated November 17, 1978.

9. Listing of output from a run of the Oil and Gas Model, dated 1/12/78.

10. Lists of data inputs to Oil and Gas Model, dated 1/12/78.

11. "Medium-Run Oil and Gas Supply Model, 1977 Update," Research

Memorandum 78-015, December 1977, Dept, of Energy, Washington,

D. C. 20461.

12. "Mid-Level Documentation," prepared by ICF to FEA under Contract

No. CO-03-60466, July 1976.

13. "Oil and Gas Mid Term Supply Model: Methodology Description,"

draft of Technical Memorandum prepared by Division of Oil and Gas

Analysis, Dept, of Energy, Washington, D. C. 20461, undated.

14. "Oil and Gas Supply Curves for the Administrator’s Annual Report,"

Technical Memorandum TM/ES/78-17, September 14, 1978, Dept, of

Energy, Washington, D. C. 20461.

15. "Project Independence Evaluation Systems (PIES) Documentation,

Vol. IV, FEA Model of Oil and Gas Supply: Data Validation and

Update," FEA/N-76/414, Federal Energy Administration, Washington,

D. C., September 1976.
16 . "Project Independence Evaluation System (PIES) Documentation,

Vol. V, Drilling Profile and Regional Allocation by Profit

Maximization," FEA/N-76/415, Federal Energy Administration,

Washington, D. C., September 1976.

17. "Project Independence Evaluation System (PIES) Documentation,

Vol. VI, Methodology for Improving the Price Sensitivities

of the PIES Oil and Gas Supply Curves," FEA/N-76/416, Federal

Energy Administration, Washington, D. C., September 1976.

18. "Project Independence Evaluation System (PIES) Documentation,

Vol. VII, Methodology for Developing More Complex Investment

and Production Profiles in the FEA Oil and Gas Model,"

FEA/N-76/417, September 1976.

19. "Project Independence Evaluation System (PIES) Documentation,

Vol. VIII, Methodology for Enabling the PIES Oil and Gas

Supply Curves to Respond to Non-Constant Prices," FEA/N-76/418,

September 1976.

20. "Project Independence Evaluation System (PIES) Documentation,

Vol. IX, Allocation of Exploratory Activity to Oil and Natural

Gas in FEA Gas Supply Model," FEA/N-76/419, September 1976.

21. "Project Independence Evaluation System (PIES) Documentation,

Vol. X, Automation of Funding Rate and Discount Rates in the

FEA Gas Supply Model," FEA/N-76/420, September 1976.

22. "Project Independence Evaluation System (PIES) Documentation,

Vol. XI, Finance Submodel for the FEA Oil and Gas Supply Model,"

FEA/N-76/421, September 1976.

23. Schanze, Jr., J. J. , "Oil and Gas Resources—Welcome to Uncertainty,

Resources: Resources for the Future , Special Issue No. 58, March

1978.

24. "U. S. Oil and Gas Supply Computer Program Documentation," National

Petroleum Council, November 1973.

25. Unpublished Notes on Oil and Gas Model Overlay Structure, undated.
NBS-114A (REV. 9-7 8)

U.S. DEPT. OF COMM. 2. Gov't Accession NoJ 2. ReeWftf* Accession No. BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO. NBS IR 80 - 1971

' IllliiBillli

4. TITLE AND SUBTITLE

First Interim Report on Model Assessment Methodology:

Documentation Assessment

5. Publication Date

7. AUTHOR(S)

Richard H. F. Jackson, Saul I. Gass, Karla L,

T.flmhprt. S. Jnel , Patsv B. Saunders

Hoffman,

8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, DC 20234

11. Contract/Grant No.

12. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street, city, state, ZIP) 13. Type of Report & Period Covered

15. SUPPLEMENTARY NOTES

| | Document describes a computer program; SF-185, FIPS Software Summary, is attached.

16. ABSTRACT A 200-word or teas factual summary of moat significant information. If document includes a significant bibliography or

literature survey, mention it here.)

report describes the results of the documentation assessment phase of a. project

to assess the Department of Energy's Midterm Oil and Gas Supply Model. The objective

here is not merely to record our conclusions about the documentation of that model,

but also to present a methodological approach to documentation assessment. These

methodological investigations have resulted in a set of guidelines which can be used

both to assist project sponsors in determining their documentation needs, and as a^

standard against which to compare existing model reports. The documentation guidelines

presented here amplify but do not alter substantively the DOE Interim Model Guidelines" of December 1978 and are organized according to levels associated with the information needs of various phases of model operation. The documentation of the

Midterm Oil and Gas Supply Model is discussed in light of these guidelines. The

guidelines are recommended for incorporation into DOE model development projects.

17. KEY WORDS (six to twelve entries; alphabetical order; capitalize only the first letter of the first key word unless a proper name;

separated by semicolons)

Documentation assessment; documentation guidelines; large-scale models; model

assessment

18. AVAILABILITY (x} Unlimited

I | For Official Distribution. Do Not Release to NTIS

| | Order From Sup. of Doc., U.S. Government Printing Office, Washington, DC

20402, SD Stock No. SN003-Q03-

(x~i Order From National Technical Information Service (NTIS), Springfield,

VA. 22161

19. SECURITY CLASS

(THIS REPORT)

UNCLASSIFIED

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

21. NO. OF

PRINTED PAGES

62

22. Price $7.00