. •

•, *.

..V'l-,

'

,V.< 'Sn' •' !
NBSIR 81-2327

MONITORING SYSTEM DEVELOPMENT:

A framework and application

Stephen D. Garrity

Experimental Technology Incentives Program

September 1981

Prepared for Experimental Technology Incentives Program
THE EXPERIMENTAL TECHNOLOGY INCENTIVES PROGRAM

The Experimental Technology Incentives Program (ETIP) of the National

Bureau of Standards pursues an understanding of the relationships between government policies and technology-based economic growth. The

pursuit of this objective is based on three premises:

• Technological change is a significant contributor to social

and economic development in the United States.

• Federal, State, and local government policies can influence

the rate and direction of technological change.

• Current understanding of this influence and its impact

on social and economic factors is incomplete.

ETIP seeks to improve public policy and the policy research process in

order to facilitate technological change in the private sector. The

program does not pursue technological change per se. Rather, its mission is to examine and experiment with government policies and practices

in order to identify and assist in the removal of government-related

barriers and to correct inherent market imperfections that impede the

innovation process.

ETIP assists other government agencies in the design and conduct of

policy experiments. Key agency decisionmakers are intimately involved

in these experiments to ensure that the results are incorporated in the

policymaking process. ETIP provides its agency partners with both

analytical assistance and funding for the experiments while it oversees

the evaluation function.

Because all government activities potentially can influence the rate and

direction of technological change, ETIP works with a wide variety of

agencies, including those that have regulatory, procurement, R+D, and

subsidy responsibilities. Programs are currently underway with the

General Services Administration, Food and Drug Administration, Veterans

Administration, Securities and Exchange Commission, Department of Energy,

Environmental Protection Agency, Occupational Safety and Health Administration, and other Federal agencies as well as various State and local

agencies.

The accompanying report was prepared from an internal study of system

developments conducted within ETIP programs. The purpose of the study

was to examine the problems of developing complex systems in the ETIP

environment and provide an approach for improved management of those

problems. Statements contained in this document represent the views

of the author and do not necessarily reflect those of ETIP or the

National Bureau of Standards.

Director

Experimental Technology Incentives Program

National Bureau of Standards

U.S. Department of Commerce
Ill

ABSTRACT

MONITORING SYSTEM DEVELOPMENT: A FRAMEWORK AND APPLICATION

Projects aimed at developing new systems where there are substantial uncertainties as to system requirements, development processes,

and ultimate ownership can present project managers with a range of

complex, unstructured problems. A process to help identify and solve

these problems in a timely, controlled manner is of central importance

to the successful conduct of a development with these circumstances.

This dissertation describes a research project devoted to examination of problems in complex system developments and to the development

of a process managers can use to deal with them. Conducted within

the Experimental Technology Incentives Program (ETIP), the research

includes analysis of several ETIP projects, a review of the system

literature, presentation of a monitoring framework to help manage

complex developments, and a brief application of the framework to

one ETIP project.

The proposed monitoring process consists of a framework of thirty

factors and a set of five functions which monitoring can serve. The

factors are divided into three main categories — design, process,

and user commitment characteristics — which reflect the general types

of problems found to be important in the ETIP environment and the
systems literature. The functions of the framework, designed to serve

both administrative and research purposes, include problem identification, strategy development, research, documentation, and dissemination. A model of how the framework might be implemented within a

project is also described.

The research makes several contributions. For ETIP, the identification of the problem and the studies of specific projects represent

the first detailed examination of system development processes within

the program. In addition, the proposed monitoring framework offers

an approach for the improvement of future development projects.

For others outside of the ETIP environment, the results may have

both administrative and research value. Other system developers are

expected to find the framework useful for improving the management

of complex system developments in their own situations. For instance,

the framework factors may suggest several new considerations important

in their projects. In addition, the functions monitoring can serve

may suggest ways that a structured, routine information gathering

process can improve their own management processes.

For researchers interested in system development processes, the

dissertation results provide several contributions. First, the identif

cation of the monitoring problem represents a more detailed examination

of the problem than has been undertaken in many parts of the literature

Second, the proposed framework offers researchers a mechanism which may
V

assist in the design and implementation of future studies. By combining

research objectives with administrative use of the framework, researchers

should be able to conduct structured investigations of system developments

in real-time rather than having to rely on the more common retrospective

approaches.

More generally, the results of the dissertation may be of value

to developers and researchers involved with a wide variety of complex

developments
VI

PREFACE

"There is nothing more difficult to plan, more doubtful of success, nor more dangerous to manage than the creation of a new system. For

the initiator has the enmity of all who would profit by the preservation of the old system and merely lukewarm defenders in those who

would gain by the new one."

Machiavelli, The Prince , 1513

(Adapted from Crandell , G. M.

Why EDP projects miscarry.

The McKinsey Quarterly, 1978,

pp. 67-74)
vii

ACKNOWLEDGEMENTS

Numerous people have been involved with the conduct of this research

both inside and outside of ETIP. I want to recognize many of them for

their support, encouragement, and comments.

First, I want to thank my dissertation chairman, Charles W.N.

Thompson, for his support, guidance, and inspiration during my research project and my work at ETIP in general. I would also like to

thank my other two committee members, Gustave J. Rath and Joseph L.

Schofer, for their support and encouragement over the years and ideas on

the final form of the dissertation.

Next, I want to thank several people within ETIP who have supported

the research over the years it has been underway. Bob Parsons, the ETIP

Director, and Daniel Fulmer, my supervisor, have both been very supportive in the final stages of my research and have given me the space to

finish it. I would also like to recognize Jordan D. Lewis, the past director of ETIP, who provided the opportunity for me to conduct the research, and Victor N. Berlin, my past supervisor in Experimental

Methods, who helped me develop ray skills in ETIP projects, provided

much guidance and encouragement for my research, and provided comments

on the final document. Bud Libraan and Roland Weiss (formerly of ETIP)

both receive ray thanks for their numerous comments and ideas on my research. Mary Mogee receives my thanks for her support and my hope that

her dissertation proceeds well and successfully. Finally, special

thanks to Madge Cooksey, my typist, who has been with me the whole way.

Thanks for sticking with it Madge, I could not have done it without
you. Best wishes for your future.

Next, I want to thank my special friend Warren Frederick. Warren

has provided immeasureable help as a close friend and colleague during

this project. The final outcome reflects a lot of his thinking. Warren,

in relation to our "beasties" contest, I still think you win hands down.

There have been many friends who have provided support to me in

the past several years during this long project. I want to thank them

for listening to me and showing me what friends truly are. My heart

felt thanks to Kathleen Flaherty, Ken Davis, Francois Alouf, Jon

Mundorff, Mary Stewart, Eileen Kahn, Donna Tobias, Rich Freeman,

Dee Seymour, Karen Stoyanoff, Nancy Remley Whiteley, Eliot Roseman,

Dave Rittenhouse, Elise Blumenthal, Lonnie and Donna Au, Jim

deBettencourt , Jane and Paul Mandel.

Most specially, I want to thank Mary Reyner for her constant and

close support and patience. Her confidence and caring has meant much

to me and has helped me though many difficult times.

Finally, I want to thank members of my family. Bill, Suzanne, and

Bill E., for their interest and encouragement. You all believed

in me and its finally finished.
IX

TABLE OF CONTENTS

ABSTRACT iii

PREFACE Vi

ACKNOWLEDGEMENTS vii

TABLE OF CONTENTS ix

LIST OF FIGURES xvii

CHAPTER 1. INTRODUCTION AND SUMMARY 1

1.1 OVERVIEW OF THIS CHAPTER 1

1.2 INITIAL DESCRIPTION OF THE PROBLEM 2

1.2.1 The Experimental Technology Incentives

Program 2

1.2.2 ETIP System Development Projects 3

1.2.3 ETIP’s System Development Problems 4

1.2.4 A Monitoring Problem 6

1.3 RESEARCH DESIGN AND SELECTED RESULTS 8

1.3*1 Objectives 8

1.3.2 Specific Tasks 9

1.3. 2.1 Monitoring Framework Development 9

1.3. 2. 2 Exploring the Usefulness of the

Framework 11

1.3.3 Classification of the Research Approach 12

1.3. 3*1 Exploratory-A Priori Dimension 12

1 . 3 . 3.Normative-Empirical Dimension 13

1 . 3 . 3 . 3 Study-Experiment Dimension l4

1.4 VALUE OF THE RESEARCH 14

1.5 A GUIDE TO THE DISSERTATION I 6

1 . 5.1 How to Read the Dissertation I 6

1 . 5.2 Chapter Summaries I 8
X

1.5. 2.1 Chapter 2 18

1.5. 2. 2 Chapter 3 18

1.5. 2. 3 Chapter 4 19

1.5. 2. 4 Chapter 5 19

1.5. 2. 5 Chapter 6 20

1.5. 2. 6 Chapter 7 20

CHAPTER 2. SYSTEM DEVELOPMENT EXPERIENCES IN THE EXPERIMENTAL

TECHNOLOGY INCENTIVES PROGRAM 21

2.1 INTRODUCTION 21

2.1.1 The Need to Review ETIP Experiences 21

2.1.2 This Chapter 23

2.2. A PLANNING SYSTEM 29

2.2.1 Background and Objectives 30

2.2.2 Project Implementation 32

2.2.3 Important Factors in System Development 34

2.3 A DISTRIBUTION SYSTEM 38

2.3.1 Background and Objectives 38

2 . 3.2 NASPO Implementation 40

2 . 3.3 NIGP Implementation 41 2 . 3.4 Project Reviews and Revisions 42

2 . 3.5 Important Factors in Systems Development 45

2.4 A DATA BASE SYSTEM 50

2.4.1 Background and Objectives 51

2.4.2 Project Implementation 54

2 . 4.3 Important Factors in System Development 60

2.5 SUMMARY AND COMMENTS 65

CHAPTER 3 . SYSTEM DEVELOPMENT PERSPECTIVES AND PROBLEMS IN

THE LITERATURE 67

3.1 INTRODUCTION 67

3.2 SYSTEM DEVELOPMENT LITERATURE 68

3 . 2.1 The Systems Literature in General 68

3 . 2.2 Different Sectors of the Systems Literature 71
3.3

A REVIEW OF SPECIFIC LITERATURE AREAS 73

3 . 3.1 Case Studies of System Developments 73

3 . 3 . 1.1 A Brief Case Study 75

3 . 3 . 1.2 A Detailed Case Study 77

3 . 3 . 1.3 An Instructive Failure 82

3 . 3.2 Important Factors in System Development 85

3 . 3 . 2.1 Methodological Differences 86

3 . 3 . 2. 2 Different Numbers of Factors 90

3 . 3 . 2. 3 Different Types of Factors 92

3 . 3 . 2. 4 Comments 95

3 . 3.3 Models for System Development 97

3 . 3 . 3.1 System Engineering Models 99

3 . 3 . 3 . 2 System Acquisition Models in

Government 102

3 . 3 . 3 . 3 Information System Models 110

3 . 3 . 3 . 4 Comments 124

3 . 3.4 System Designs 128

3 . 3 . 4.1 General Issues in System Design 129

3 . 3 . 4. 2 System Design Examples 134

3 . 3.5 Conclusion 139

3.4 KEY PROBLEMS IN THE SYSTEM DEVELOPMENT LITERATURE 140

3 . 4.1 Introduction 140

3 . 4.2 The Multi-faceted Nature of Development 141

3 . 4. 2.1 Limited Perspectives in the Literature l42

3 . 4. 2. 2 A More Comprehensive View 145

3 . 4.3 Time Varying Qualities of System Development ..147

3 . 4. 3.1 Time Perspectives in the Literature ..148

3 . 4. 3 . 2 A Dynamic Perspective Needed 151

3 . 4.4 User Commitment During Development 153

3 . 4.

4.1

Commitment Perspectives in the

Literature 155
3. 4. 4.

A More Comprehensive View of User

Commitment 158

3.4.5 Calls for Research in the Systems Literature ..159

3. 4. 5.1 Research Needed on the Management of

Developments l60

3. 4. 5. 2 The Need for Management Frameworks ...163

3. 4. 5. 3 A Need for Empirical Evidence 165

3. 4. 5. 4 An Approach to Meeting These Needs ...166

3.4.6 Summary and Conclusion 167

3.5 CONCLUSION 171

CHAPTER 4. RESEARCH DESIGN 173

4.1 INTRODUCTION 173

4.2 DEVELOPMENT OF THE MONITORING FRAMEWORK 174

4.2.1 Identifying the Problem Area 175

4.2.2 Reviewing the Literature and Selecting

Important Development Factors 178

4.2.3 Redefinition of the Monitoring Problem l82

4.2.4 Refining the Framework Factors 185

4.2.5 Monitoring as Seen in the Literature 188

4.2.6 The Resulting Monitoring Framework 191

4. 2. 6.1 Monitoring Framework Factors 191

4. 2. 6. 2 Using the Monitoring Framework 195

4.2.7 A Guide to Chapters on Framework

Development 196

4.3 EXPLORING THE USEFULNESS OF THE FRAMEWORK 197

4.3.1 Introduction 197

4.3.2 Application to On-Going System

Developments 198

4 . 3.3 An Initial Retrospective Test 199

4 . 3.4 A Second Retrospective Application 201

4. 3 . 4.1 Revising the Application Design 201

4. 3 . 4. 2 Developing a Brief Chronology

of the Case 203

4. 3 . 4. 3 Applying the Framework 205

4. 3. 4.

Limitations and Biases of the

Approach 208

4.4 CONCLUSION 210
CHAPTER 5. THE FACTORS AND FUNCTIONS OF A FRAMEWORK FOR

MONITORING SYSTEM DEVELOPMENTS 212

5.1 INTRODUCTION 212

5.1.1 The Monitoring Factors 213

5.1.2 The Monitoring Functions 215

5.1.3 Organization of the Chapter 222

5.2 DESIGN CHARACTERISTICS (FACTORS 1-12) 225

5.2.1 Introduction 225

5.2.2 Operation and Performance Factors 227

5. 2. 2.1 Factor 1 —Response Time 228

5. 2. 2.Factor 2 —Quality 230

5. 2. 2.Factor 3 —Cost of Operation 233

5.2.2.

Factor 4 —Input/Output Operations ....235

5. 2. 2.

Factor 5 —Interconnection of

Subsystems 238

5.2.3 Boundary Factors 240

5. 2. 3.1 Factor 6 —Capabilities/Limitations/

Expectations 242

5. 2. 3.Factor 7 —User Groups and Their

Interrelationships 246

5. 2. 3.Factor 8 —Interfaces with Other Systems and Organizations 249

5.2.4 Adaptation Factors 251

5.2. 4.1 Factor 9 —Flexible Specifications ,...253

5.2. 4.2 Factor 10 —Matching the System

to the User 257

5.2. 4.3 Factor 11 —Novelty of the Design 264

5. 2. 4.

Factor 12 —Evaluation and Updating ...269

5.3 PROCESS CHARACTERISTICS (FACTORS 13-24) ,...272

5 . 3.1 Introduction 272

5 . 3.2 Task Structure Factors 275

5 . 3.2.1 Factor 13 —Task Size 277

5 . 3 . 2.

Factor 14—Task Priorities 280

5 . 3.3 Team Personnel Factors 285

5. 3 .

3.1

Factor 15— Skills 287
5. 3. 3. 2 Factor 16 —Turnover 290

5. 3.3. 3 Factor 17 —Commitment 293

5 . 3 .^ Project Control Factors 298

5 . 3

.

^.! Factor I 8 —Organization and

Responsibilities 300

5 . 3

.

^*2 Factor 19 —Decision Points and

Milestones 309

5 . 3 . 3 Factor 20 —Reports and Reviews 316

5 . 3 • 5 Interaction with User Environment Factors 323

5 . 3 . 5.1 Factor 21 —User Involvement 325

5. 3 . 5. 2 Factor 22 —Problem Identification ....329

5.3«5.3 Factor 23 —Testing 335

5. 3 . 5. 4 Factor 24—Transfer 341

5.4 USER COMMITMENT CHARACTERISTICS (FACTORS 25-30) 347

5.4.1 Introduction 347

5.4.2 System Use Factors 349

5. 4.2.1 Factor 25 —Applications 351

5. 4. 2. 2 Factor 26 —Consequent Actions 354

5. 4 . 2.3 Factor 27 —Extent of Use 356

5 . 4.3 System Support Factors 361

5. 4 . 3.1 Factor 28 —System Champions 363

5. 4. 3 . 2 Factor 29 —Resource Commitments 367

5. 4. 3 . 3 Factor 30 —Changes in the User

Organization 372

5.5 THE MONITORING FUNCTIONS 375

5 . 5.1 Introduction 375

5 . 5.2 Function #1 —Problem Identification 376

5 . 5.3 Function #2 —Strategy Development 376

5 . 5.4 Function #3 —Research 377

5 . 5.5 Function #4—Documentation 378

5 . 5.6 Function #5 —Dissemination 380

5 . 5.7 A Model for Applying the Framework 382

5.6

CONCLUSION 389
XV

CHAPTER 6. AN APPLICATION OF THE FRAMEWORK TO A SYSTEM

DEVELOPMENT 391

6.1 INTRODUCTION 391

6.2 A BRIEF CHRONOLOGY OF THE EVALUATION SYSTEM

DEVELOPMENT 392

6.2.1 Introduction 392

6.2.2 Background and Objectives of the Project 397

6. 2. 2.1 The Emergence of an Evaluation

Program in ETIP 397

6. 2. 2.

Evaluation in the ETIP Procurement

Program 398

6. 2. 2.

Developing a Work Statement 399

6. 2. 2.

Awarding the Contracts 401

6.2.3 Phase 1 402

6. 2. 3.1 The First Six Months of Phase 1,

July-December 1976 402

6. 2. 3.

The Second Six Months of Phase 1,

January- June 1977 404

6. 2. 3.

The End of Phase 1, JulySeptember 1977 407

6.2.4 Phase 2 409

6.2. 4.1 The First Six Months of Phase 2,

September 1977-March 1978 409

6. 2. 4.

The Second Six Months of Phase 2,

April-September 1978 4ll

6. 2. 4.

The Last Six Months of Phase 2,

October 1978-March 1979 413

6.2.5 The Remainder of Phase 3> AprilDecember 1979 4l4

6.2.6 Conclusion 4l6

6.3 APPLICATION OF THE MONITORING FRAMEWORK TO ONE

SELECTED PERIOD OF THE SYSTEM DEVELOPMENT 4l6

6.3.1 Introduction 4l6

6.3.2 Reports and Reviews, July-December 1976 ^18

6. 3. 2.1 Introduction and Summary 4l8

6. 3. 2.

Establishing the System for

Reports and Reviews 419
6. 3. 2.

Problems with the Reports and

Reviews System 421

6. 3 -2.

Effects on Other Areas of the

Project 423

6. 3. 2.Resolving the Problems with Reports

and Reviews 434

6. 3 .2.

How the Monitoring Framework Could

Have Helped 436

6.4 CONCLUSION 439

CHAPTER 7. CONCLUSIONS AND RECOMMENDATIONS FOR FURTHER

RESEARCH 440

7.1 AN OVERVIEW OF RESULTS 440

7.2 CONTRIBUTIONS OF THE RESEARCH 444

7.2.1 Increase in Confidence 444

7. 2. 1.1 Definition of the Problem 445

7. 2. 1.2 Development of the Monitoring

Framework 448

7.2.2 Utility of the Results 449

7.3 EXTENSIONS OF THE RESEARCH 452

7 . 3.1 Improving Framework Design 453

7 . 3.2 Testing the Framework 455

REFERENCE NOTES 457

BIBLIOGRAPHY 466

APPENDIX A Background Materials on the Experimental Technology

Incentives Program 475

APPENDIX B A Manual for Designing and Implementing A Process to

Monitor Complex System Developments 519

APPENDIX C Letters from Reviewers of the Dissertation 585
xvii

LIST OF FIGURE'=^

Figure

Number Title Page

2.1 Chronology of Public Buildings ServiceETIP Project to Develop Space Acquisition

Planning Model 24

2.2 Chronology of ETIP Project with NIGP

and NASPO to Develop State and Local

Agency Procurement 25

2.3 Chronology of ETIP Project with Small

Business Administration to Develop a System to Support Set-Aside Process 27

4.1 Decomposition of a Development Model 187

4.2 The Monitoring Framework 192

5.1 The Monitoring Framework 2l6

5.2 Brief Descriptions of the Monitoring

Factors 217

5.3 A General Model for Applying the

Monitoring Framevrork 382

5.4 Problem Solving Using the Framework 386

5.5 Strategy Development Using the Framework 388

6.1 Brief Chronology of the ETIP Project to

Develop An Evaluation System for the

Procurement Program 393

6.2 Major Events and Phases of the Evaluation

System Development on a Time Line 395

6.3 Relationship of Activities Concerning

Reports and Reviews to Cther Project Areas.... 425

7 . 1 The Monitoring Framework 443
n

?:5^

'"p

;y

. '. fv'.i

r'lir.’cmm

: ,j,

'. -K.''.

'

'V^!' S> * -< • -•<» *

.' • i -'iltoj

m

rm^

’.•'

’* -'Ji

mgmi

M

smmrnm
1

CHAPTER 1

INTRODUCTION AND SUMMARY

1.1 OVERVIEW OF THIS CHAPTER

The development of systems where there are substantial uncertainties

as to system requirements, development processes, and ultimate ownership

can present managers with complex, unstructured problems. Recent developments of this nature in the Experimental Technology Incentives Program

demonstrate that managers need a mechanism to handle these problems in

a manner which promotes timely awareness and resolution of the problems

as well as increased attention to both their short and long term implications for strategy.

This dissertation is devoted to an examination of the types of problems which can arise in complex developments and to the development of

a monitoring mechanism managers can use to more effectively control them.

The specific focus is on system developments conducted within the Experimental Technology Incentives Program.

The following sections of Chapter 1 outline the activities, results,

and contributions of the research into those problems. References are

made to the chapters where more specific details are presented and a

directory to the dissertation is also provided.
1.2 INITIAL DESCRIPTION OF THE PROBLEM

1.2.1 The Experimental Technology Incentives Program

The research being reported in this dissertation took place in

the Experimental Technology Incentives Program (ETIP) of the National

Bureau of Standards. Created in 1972, ETIP's goal is to develop a

better understanding of the relationships between government policies

and technology-based economic growth. This goal is pursued through

the conduct of studies and experiments with government agencies whose

policies have a linkage to technological change in the private sector.

In recent years, ETIP has focused on regulatory, economic, R&D, and

procurement policies.

The primary research approach used by ETIP is that of administrative experimentation (Campbell, 1969; Thompson and Rath, 1974).

ETIP and an agency join together as partners in projects where there

is a shared interest in various elements of a policy-technology relationship. Where a policy change offers the potential to learn more

about the elements and linkages of a specific policy-technology relationship, an experiment is then defined. In the research role,

ETIP offers funding and analytical support for all phases of an experiment, including experiment development, design, implementation, and

evaluation. In the administrative role, the respective agency offers

1 More specific information on ETIP can be found in APPENDIX A.
3

the action setting, policymaking power, and resources to help support

the experiment and its evaluation. Using this approach, ETIP and

the partner agency learn more about agency practices and impacts in

a manner which meets the interest of both participants.

1.2.2 ETIP System Development Projects

Some ETIP projects involve the development of information systems

as part of the experimentation process. For example, an agency partner may want to improve the process by which it implements a policy.

In this case, a system constitutes the primary objective of a project

and it is expected to lead to an on-going means for studying a policytechnology relationship. In other situations, development of a system

may be tied to a pending policy change where a process is needed to

monitor and evaluate the effects of the change. Some recent examples

of these types of projects have included a data base system, an impact

monitoring system, an information distribution system, and a planning

system.

One particular systems project involved the development of an

evaluation system that was to be used in conjunction with the experiments underway in the Procurement Program of ETIP. This project was

initiated by ETIP over the need to develop a more formal program of

evaluation for the procurement experiments that could meet agency

2 Several of these are described in more detail in Chapter 2 as part

of this research.
It was

3 and ETIP information needs over an extended period of time,

expected that, with the continuation of the Procurement Program, the

system would eventually be institutionalized within one or more agencies involved with procurement experiments.

While this system development was similar in some respects to

other ETIP experiences, it also represented a fairly new approach

in ETIP that presented project managers with several novel problems.

These are further explained below.

1.2.3 ETIP's System Development Problems

The evaluation system project discussed above presented ETIP

project managers with several special problems. First, there were

substantial uncertainties as to the requirements for the system.

These were due to several circumstances:

• The Procurement Program was highly fluid in that the number

and type of experiments (and agency partners) were variable.

• There were numerous objectives for evaluation from specific

and immediate (e.g., what happened in an experiment) to

broad and long term (e.g., the implications of a series

of experiments for policy changes in the agency).

• There were numerous stakeholders in the procurement work,

including Federal, State, and local partner agencies, and

related public and private institutions with a stake in

procurement policies.

3 More background on the initiation of this project is presented

in Chapter 6.
Generally, the requirements for the evaluation system could be characterized as broad, changing, and complex. It was unclear what evaluation

would be needed as well as what system design would be needed to conduct evaluations in the long term.

Second, there were also substantial uncertainties in the system

development process. A development of this nature had not been conducted by ETIP before and similar experiences in other environments

were unknown or nonexistent at that time. In addition, the development team was new and had to be integrated into the already established

relationships between the Procurement Program and its partner agencies. There were two aspects to this problem. First, the project

managers in ETIP were new and had to establish ties with the Procurement Program. Second, as contractor support was acquired to help

in the development, a third set of personnel outside of ETIP had to

4

be included as another party to the procurement experiments. Those

characteristics combined with the aforementioned uncertainties over

system requirements necessitated a flexible, evolving approach to

the development as both relationships and knowledge of evaluation

needs increased.

Finally, the project was further complicated by the uncertainties

over eventual ownership of the evaluation system. While ETIP was

4 ,Acquiring contractor support for the project was itself a novel

problem. See Chapter 6 for more discussion of the procurement and Libman (1980) for a complete examination.
6

fully supporting the development and expected to derive information

from the system in the long run, it was not expected nor planned to

be the primary user and owner. One of ETIP’s long term goals was

to institutionalize the experimentation process in agencies which

found it to be a value in their continuing development and implementation of policies. The system development was thus planned with this

goal in mind and an objective was established to turn-key the system

after its development. However, when the project was initiated, it

was unclear who the eventual owners would be and what form the institutionalization process would take.

1.2.4 A Monitoring Problem

ETIP project managers were faced with a system development in

which there was substantial uncertainty in system design, development

process, and ultimate ownership. As the project began, these uncertainties, and the numerous activities which were underway, presented managers

(one of which was the writer) with a need for a mechanism for project

control. Day-to-day problems and progress often consumed the time

and attention of managers at the expense of maintaining a more overall

perspective on the direction of the project. Problems were difficult

to anticipate in advance and problem solving often had to be conducted

under pressure and with limited examination of both short and long

term implications. In addition, strategy was being developed incrementally, often in response to short term problems, leaving some uncertainties as to whether the project was on course and leading towards
a successful development.

The need for a control mechanism was the earliest idea for the

research project reported in this dissertation. At first, the problem

was characterized as one of the need for a strategy model. Project

managers were seen as needing an ideal model on which to judge whether

the project was on course and to better anticipate and plan the specific activities that should occur. It was also viewed as a means of

anticipating problems and providing a longer term perspective to their

solution.

Various initial studies of other ETIP projects and the systems

literature were then conducted to develop a strategy model. It was

expected that these other experiences would provide "proven" elements

of strategy that would assure the chances of success for ETIP managers

in the evaluation system project.

However, a model for development strategy was not identified

in these sources that could be readily adapted into the particular

ETIP situation. If anything, it was clear that no one model existed

or could be developed in detail that would reflect the changing and

complex circumstances the development presented. It thus came to

be seen in the research that the real need of the management team

was for some way to develop strategy over time, based on a more complete awareness of project activities as they unfolded.
The problem of concern in the dissertation therefore was changed

from one of developing a model to one of developing a mechanism which

managers could use to monitor progress. Further research was aimed

at developing this mechanism and applying it to the ETIP environment.

1.3 RESEARCH DESIGN AND SELECTED RESULTS

This section summarizes the research design which was used to

develop and explore the usefulness of the monitoring framework. The

discussion includes a review of the objectives and the specific tasks

which occurred, and an overall classification of the research design.

Presenting this information has been a problem to some extent

since the research was exploratory and involved the concurrent activities of examining the monitoring problem and developing a specific

monitoring approach. This section thus provides an overview of what

was done without too much regard to the order with which activities

occurred. A more detailed presentation of the actual process is presented in Chapter 4.

1 . 3.1 Objectives

With the initial problem identified, as described above (Section

1.1), there were two main objectives for the research described in

this dissertation. These included:
9

1. To develop a framework for monitoring system developments

that would provide managers with a basis for assessing and

revising project strategy, and

2. To explore the usefulness of the framework as a management

tool in actual system projects.

Most of the emphasis in the research has been devoted to

framework development. This was primarily due to the need further

refine the problem of monitoring and then to devise a mechanism as

a proposed approach. The extensive time consumed in this process

eventually eliminated the opportunity to explore the usefulness of

the approach in real-time in an actual ETIP case. A retrospective

application was then conducted to at least initially examine the

merits of the approach.

1.3.2 Specific Tasks

This section summarizes the various elements of the research

design which occurred under each main objective. More specific details and results of these tasks are provided elsewhere in the dissertation, and references to the appropriate chapters are made.

1.3. 2.1 Monitoring Framework Development

The first objective of the research was to develop a monitoring

framework that ETIP managers could use to assess and revise strategy

for a system development. Work within this area consisted of two

main components — that of further refining the monitoring problem
while simultaneously developing a specific monitoring approach. Overall, there were five tasks which were conducted in this portion of

the research. These included;

1. Analyzing several ETIP system developments.

Three ETIP cases were examined for factors important in

system development (see Chapter 2). Examples from the cases

were later used to illustrate the specific factors of the

monitoring framework (Chapter 5).

2. Reviewing the systems literature.

The systems literature was also reviewed to identify more

factors important in developments (Chapter 3)* Problems

and gaps found in this review influenced framework design.

Later, the literature was scanned more closely for ideas

on monitoring (Chapter 4).

3. Reviewing ETIP management needs.

The ideas generated by the reviews of ETIP projects and

the literature were compared against the needs of managers

in the specific case which initiated the research (Chapter 4).

4. Organizing factors into a monitoring framework.

The factors identified in ETIP projects and the literature

were refined and organized into a framework structure.

Thirty factors were identified in three main categories —design, process, and user commitment characteristics

(Chapters 4 and 5).

5. Developing the monitoring functions.

A set of procedures to accompany the factors was also developed. Five main functions for monitoring were identified — problem identification, strategy development, research,

documentation, and dissemination. A general model of how

the framework might be applied was also developed (Chapters

4 and 5 )
11

1.3. 2. 2 Exploring the Usefulness of the Framework

The second objective of the research was to explore the usefulness of the framework as a management tool in actual developments.

Specifically, it was desired to examine whether the framework could

assist managers with problem identification, strategy development,

or the documentation and dissemination of progress in the course of

conducting a development.

While the original expectation was that one on-going project in

ETIP would be used (the evaluation system project), the duration of

framework development activities eliminated this opportunity. A less

effective retrospective approach was then designed and applied to the

completed evaluation system development.

Three tasks were conducted in the retrospective application of

the framework to the ETIP evaluation system case. These included:

1. Developing a case history of the project.

A case study of the project was conducted to familiarize

the writer with events and to identify selected periods

where the framework might be applied (Chapter 4). One period

was selected. A brief chronology of the project was written

for the dissertation (Chapter 6 ).

2. Mapping the framework onto the project.

The preliminary task of the actual framework application

was to map the framework factors onto project events during

the period selected earlier (Chapter 4). This demonstrated

that the factors were relevant to the case and prepared

the way for exploring how the framework might have helped

managers (Chapter 6).
12

3 . Examining how the framework might have helped managers.

The mapping was used to show that use of the framework might

have helped managers identify a critical problem of the

period (Chapter 6 ).

1.3.3 Classification of the Research Approach

The research approach taken to meet the objectives described

above can be classified according to three research dimensions developed

by Thompson and Rath ( 1974 ) —Exploratory-A Priori, Normative-Empirical, and Study-Experiment. Each of these dimensions is discussed

below.

1.3. 3.1 Exploratory-A Priori Dimension

The Exploratory-A Priori dimension measures the extent to which

the researcher predetermines the effect of new data on his results.

Exploratory situations are those characterized by many unknowns in

data, methods, and results while a priori situations tend to be more

predictable, controllable, and reaffirming.

On this dimension, the research was exploratory. There was not

much orderliness to the problem at the beginning of the effort (Roseman,

1978, pp. 1 - 9 ). Concepts needed to be identified and developed in

order to provide structure to the problem. While sources in ETIP

and the systems literature contained many elements related to the

problem, a detailed recognition of its dimensions and components was
not uncovered. The dissertation has both identified a problem in

more detail and proposed a specific approach where neither has been

explicitly recognized or developed previously.

1.3. 3. 2 Normative-Empirical Dimension

The Normative-Empirical dimension measures the extent to which

the researcher obtains new data directly from the phenomena under

study (i.e., system developments in this case).

On this dimension, the research was somewhere in the middle.

Ideas, models, and concepts related to system development have been

used to help define the problem and the framework, thus the research

is somewhat normative. In addition, the primary contributions of

the research are ideas and not empirical data. On the other hand,

some more direct experiences have also been a part of framework development and its preliminary application. Several system projects in

ETIP have been examined and used to illustrate factors in the framework. An additional case has also been used to explore the usefulness

of the framework in an actual case. Although this application relies

on a retrospective analysis of project records, the writer's experience with the project as it unfolded also adds a more direct connection to the specific events. These cases thus add an empirical element to the research, although a more thorough testing directly in

a project is still needed.
14

1.3. 3.

Study-Experiment Dimension

Finally, the Study-Experiment dimension measures the extent to

which the researcher manipulates the phenomena under investigation.

On this dimension, the research is more of a study than an experiment. The ideas and experiences of others as reported in ETIP records

and the systems literature have been the primary source of material

on the system development process. Problem definition, framework

development, and exploration of framework usefulness have thus relied

to a large extent on sources not directly controlled by the writer

as part of the dissertation research.

However, it also should be noted that the research has closely

interacted with the writer's experience in a system development (the

evaluation system project). The experience in this project provided

the impetus to begin the research as well as a continuing benchmark

against which to "test" understanding of the problem and ideas on

the monitoring framework. While not an experiment in the formal sense,

this interaction represents an element of control which likely has

affected the implementation of this research.

1.4 VALUE OF THE RESEARCH

The dissertation makes several contributions of value to both

administrators of system developments and researchers of system develop-
ment processes (Chapter 7). For ETIP, the identification of the problem and the studies of specific projects represent the first detailed

examination of system development processes within the program. In

addition, the proposed framework offers an approach for the improved

management of future development projects.

Other system developers are expected to find the framework useful

for improving the management of complex system developments in their

own situations. First, developers may find the factors of the framework useful for suggesting new considerations important in their projects. Second, developers may find the functions monitoring can serve

useful in improving their own management practices. The proposed

framework may be able to supplement existing practices for the identification of project management problems, the development and revision

of strategy, or the documentation and dissemination of progress for

decisionmaking on project implementation.

For researchers interested in system development processes, the

dissertation results are of value in several areas. First, the identif

cation of the monitoring problem represents a more detailed examination of the problem than has been undertaken in many parts of the

literature. Second, the proposed framework offers researchers a raechan

ism which may assist in the design and implementation of future studies

By combining research objectives with administrative use of the framework, researchers should be able to conduct structured investigations
16

of system developments in real-time, rather than continuing to rely

on case studies and retrospective approaches.

Finally, the results may be of value to administrators and researchers of a variety of complex developments. While the framework is

specificallly designed for the ETIP environment and its information

system developments, its components are based on ideas and experiences

associated with a wide variety of systems. The dissertation may thus

offer guidance for other kinds of developments, or at least promote

an increased awareness of the similarity of problems often encountered

in distinctly different types of developments.

1.5 A GUIDE TO THE DISSERTATION

1.5.1 How to Read the Dissertation

The organization of the dissertation has generally followed the

progression of research activities as they occurred. For example,

Chapters 2 and 3 review ETIP projects and the systems literature which

were used to help define the problem and support development of the

monitoring framework. Chapters 5 and 6 present the resulting framework and explore its usefulness in an additional ETIP project.

Depending on the interests of the reader, there can be several

approaches to reading the dissertation. Most readers may find it

best to review Chapter 1 first in order to obtain an overview of the
dissertation. Chapter summaries are provided below to further identify contents in each chapter.

Researchers of system development processes may find several

components of the dissertation useful for their work. Therefore,

researchers may want to scan Chapter 4 next to examine the monitoring

problem which has been identified and the research process which has

been used. Following this, researchers should turn to Chapter 3 and

examine how the systems literature has been used to define the problem

and to develop an approach to project monitoring. Some researchers

may also find the ETIP case studies of Chapter 2 interesting as detailed descriptions of actual projects and as models for the documentation of future projects. Finally, researchers should examine the

monitoring framework of Chapter 5 and a shortened "how to do it" manual

located in APPENDIX B. They should find the factors of the framework

which relate to their current research. Researchers should then determine whether the framework offers an approach for structuring future

research projects and gaining access to actual projects for the conduct of research.

Managers of system development projects will most likely find

the monitoring framework the primary contribution of the dissertation.

Thus, they should turn to Chapter 5 directly and examine the monitoring factors and the monitoring functions which are proposed. A preliminary draft of a "how to do it" manual has also been prepared from

Chapter 5 and included in APPENDIX B. System developers may find
18

this shortened version more appropriate for their review. Finally,

if further ideas on the application of the framework to a project

are needed, developers may want to scan Chapter 6 to examine how the

framework might have been used in an ETIP project.

1.5.2 Chapter Summaries

1.5. 2.1 Chapter 2

This chapter reviews three ETIP projects which involved system

developments. The reviews were initially conducted to develop a list

of factors important in an ETIP development, thus documenting the

background knowledge available in the program. Examples from these

cases are later used in Chapter 5 to illustrate factors in the monitoring framework.

1.5. 2. 2 Chapter 3

This chapter reviews the systems literature which was also consulted in the pursuit of factors important in system development.

The chapter is divided into two main parts. The first part provides

an overview of the systems literature in general to demonstrate the

types of materials studied. This portion also includes several examples of those materials which are later referenced in support of
the framework factors identified in Chapter 5. The second part of

the chapter reviews problems and gaps in knowledge in the literature

which were relevant to and influenced the development of the monitor

ing problem and framework. The items are tied to the respective

actions taken in the research.

1.5. 2. 3 Chapter 4

This chapter describes the research design as it unfolded, from

development of the problem to the exploration of the usefulness of

the framework in a specific project. References are made in the

discussion to other chapters which resulted during the process.

1.5. 2. 4 Chapter 5

This chapter presents the monitoring framework which has resulted

from the research. The thirty factors in the framework are described

and illustrated with examples from the three ETIP projects discussed

in Chapter 2. Also, the five functions monitoring can serve in a

system development are described. A model of how the framework might

5 be implemented in a project concludes the chapter.

A shortened ”hov; to do it" manual for applying the framework has

been prepared from Chapter 5 and is included in APPENDIX B. The manual

is a preliminary version and essentially summarizes materials presented

in Chapter 5.
1.5. 2. 5 Chapter 6

This chapter presents the retrospective application of the framework to a specific ETIP case. The first portion of the chapter presents a brief chronology to familiarize readers with the case. The

second portion of the chapter then shows how the framework might have

helped managers identify a critical problem during one selected period

of the project.

1.5. 2.Chapter 7

This chapter reviews the results of the research conducted and

the contributions made. In addition, some extensions to the research

are discussed.
21

CHAPTER 2

SYSTEM DEVELOPMENT EXPERIENCES IN THE

EXPERIMENTAL TECHNOLOGY INCENTIVES PROGRAM

2.1 INTRODUCTION

2.1.1 The Need to Review ETIP Experiences

As described earlier in Chapter 1, this dissertation evolved

out of a need for guidance in a specific system development in the

Experimental Technology Incentives Program (ETIP). ^ It was found

that system developers needed a device to help them monitor and

control the numerous, evolving activities and problems of the project.

One of the first tasks in the formation of the monitoring device

was to examine the guidance available in other ETIP projects which

2 involved similar information system developments. Three projects

were selected for in-depth study, including:

This was the project to develop an evaluation system for the

ETIP Procurement Program. See Chapter 6 for more details on the

project.

2

ETIP had conducted a number of similar projects before. Since

they were completed already or nearly completed, it was expected

that they might provide some guidance for the management of the newer evaluation system development.
• A data base system for the Small Business Administration

(SBA)

• A planning system for the Public Buildings Service (PBS).

• A procurement information system for state and local government agencies.

These three were used because:

• Each project had the direct or implicit objective of develop

ing a system which would become institutionalized within

an organization (similar to the evaluation system project).

• The writer was assigned to review or monitor most of the

projects.

• The projects were either completed or nearly completed,

making it possible to study a project from the beginning

to the end.

• The projects were fairly well documented, and where re- cords were weak, contact with original participants was still feasible.

Each project was reviewed in detail to determine what had happened and what key factors had influenced progress. Project documents were reviewed and many of the principal participants were

interviewed. In some cases, the writer's notes and observations

in the conduct of ETIP business were used as sources of information

for the case studies.

Important factors in development from each case were then collected and used to help develop a preliminary monitoring device

for ETIP managers in the evaluation system project. Later, examples

from each case were used to illustrate the factors in the monitoring

framework (see Chapter 5).
23

2.1.2 This Chapter

This chapter presents some of the results from the three case

studies to help set the stage for their use in later chapters.

Specifically, this chapter provides the general history on each

project and the major lessons which were found relative to their

conduct. Examples from these projects are used later in Chapter

5 to illustrate the factors important for monitoring. Readers may

also find this chapter helpful in providing an introduction to the

ETIP environment and the systems of interest in this dissertation.

Each case is presented by itself and contains several sections.

First, project background and objectives are described in order

to show how the work was initiated. These sections are followed

by others which discuss implementation. A final section in each

case describes the major factors found to be important to project

success. Major events in each project are summarized in Figures

2.1, 2.2, and 2.3 for easy reference. Major lessons learned are

summarized at the end of the chapter.

Readers will note several things about the case descriptions.

First, they tend to be brief and highlight only major issues. This

was done in order to make the cases simple to understand and more

easily usable in Chapter 5, where they are used to illustrate fac-
24

Date

March 1974

May 1974

July 1974

September 1974

October 1974

April 1975

December 1976

July 1977

September 1977

November 1977

February 1979

May 1979

February 1980

FIGURE 2.1

Event

Public Buildings Service (PBS) indicates

interest in a project to ETIP and sends

a preliminary plan.

ETIP issues a project plan and obligates

resources. Project is to develop a space

acquisition model and procedures for PBS.

Prime contractor and subcontractors ac- quired to help build new system.

Prime contractor submits proposal to PBS

to expand system development.

Contract revised and extended to provide

for more extensive modeling and integration with PBS processes.

Preliminary model prepared for PBS review.

Basic model procedures completed.

Model computerized, first demonstration

run.

Prime contractor completes project and

issues final report. Model in operation

in Special Studies unit of PBS.

ETIP begins review to determine evaluation

needs. A number of applications reported

by Special Studies unit.

The General Accounting Office issues re- port on the model and suggests several

improvements

ETIP issues report (Lawrence and Garrity)

on model development and lessons learned

for Procurement Program. Model still

in operation in Special Studies unit, but

not yet permanently institutionalized.

Project account closed by ETIP.

CHRONOLOGY OF PUBLIC BUILDINGS SERVICE - ETIP

PROJECT TO DEVELOP SPACE ACQUISITION PLANNING

MODEL
Date Event

Early 197^

June 1974

September 1974

November 1975

December 1975

February 1976

April 1976

June 1976

September 1976

October 1976

November 1976

FIGURE 2.2

ETIP discussions underway with Small Business Administration (SBA) for a joint

project.

ETIP issues plan to develop a data base

system with SBA. Plan approved by

Department of Commerce.

SBA drafting a request for proposals for

contractor support to develop the system.

Contract for development awarded. Work

begins on six main tasks.

Task 1 completed: a review of set-aside

cases in SBA.

Task 2 completed: a preliminary plan

of system and its operation to support

set-aside cases.

Task 3 completed: specific system design

(manual system) and sites for preliminary

field testing identified.

Task 4 completed: design of field tests.

SBA administrators briefed on tests.

Field workshops planned.

Task 5 completed: preliminary field tests

Eight cases obtained, system accuracy

and reliability confirmed. Operational

testing phase started for manual system.

Several workshops held. Some opposition

to using system arises in west coast site.

ETIP begins new strategy to coordinate

NASPO and NIGP procurement package develop

ments . Proposes new joint schedule. NIGP

issues Phase 1 report.

CHRONOLOGY OF ETIP PROJECT WITH NIGP AND NASPO

TO DEVELOP STATE AND LOCAL AGENCY PROCUREMENT

INFORMATION SYSTEM
26

Date Event

December 1976 ETIP plans review process for both con- tracts to determine whether they should

be renewed at end of Phase 2 in June 1977

February 1977 ETIP requests special reports from NASPO

and NIGP to answer several questions in

continuing review.

March 1977 ETIP, NASPO, and NIGP meet to review progress of Phases 1 and 2.

April 1977 ETIP decides to terminate NASPO project

after June. NIGP project extended at

no cost to continue work on promising

procurement information packages.

June 1977 NASPO project ends. NASPO issues final

report. NIGP projected extended for one

year.

June 1978 NIGP project ends (Phase 3 not funded).

January 1977 Workshops planned on east coast to build

track record of use.

April 1977 SBA begins discussion with one agency

to use system in pending procurements

involving set-asides.

FIGURE 2.2 (Continued)
27

Date

June 197^

November 1974

January 1975

April 1975

June 1975

July 1975

August 1975

October 1975

November 1975

July 1977

Event

Plan for state procurement information

system development by the National Association of State Procurement Officials (NASPO)

issued by ETIP and approved by Department

of Commerce,

Plan for local agency procurement information system development by the National

Institute of Government Purchasing (NIGP)

issued by ETIP and approved by Department

of Commerce.

NASPO project begins Phase 1. Information

needs examined and task forces formed

to develop information packages.

NASPO reports task forces underway but

problems with obtaining time from volunteer participants.

NIGP project begins Phase 1. Information

needs to be identified.

ETIP organizes a meeting with NIGP and

NASPO to discuss information system development. NASPO task forces working slowly.

ETIP outlines problems with NASPO project,

referring to lack of progress on work

statement items.

NASPO issues new work plan for rest of

Phase 1.

NASPO begins to distribute procurement

information packages from several task

forces. NIGP decides to develop packages

in-house instead of with a contractor.

Computerized version of system completed

(with an added data base) and demonstrated

in the agency. Test successful and applications started.

FIGURE 2.3 CPiRONOLOGY OF ETIP PROJECT WITH SMALL BUSINESS

ADMINISTRATION TO DEVELOP A SYSTEM TO SUPPORT

SET-ASIDE PROCESS
Date Event

September 1977 Task 6 completed: operational testing

phase. Ten manual tests and five computerized tests completed. SBA decides

not to fund operation of system further

due to lack of demand for use. SBA

awards independent contract to develop

more comprehensive system for wider range

of set-asides.

November 1977 Contractor issues final report. Concludes

computerized version of system with data

base best approach to supporting set-aside

processes.

June 1976 Phase 1 of NASPO and NIGP projects ends.

Phase 2 begins for both (12 months).

NIGP selects l8 product areas for potential procurement information packages.

August 1976 NASPO issues Phase 1 report. Concludes

that package development more complex

than expected and proposes refocusing

project on development of an institution

to work with problems in longer term.

FIGURE 2.3 (Continued)
29

tors in the monitoring framework which has been developed. Second,

readers will note that the illustrations in Chapter 5 often exceed

the details provided in this chapter. This is partly due to the

fact that Chapter 2 was written much before Chapter 5 and partly

due to the desire to leave the intricate, intertwined details out

of Chapter 2.

It should also be noted that work on Chapter 5 caused some refinements in these case descriptions after they were written. For example, in the PBS case below, readers will find that the speed and

versatility of the computer system was one factor favorably influencing user acceptance of the system. Subsequent refinements of

the monitoring framework as shown in Chapter 5 later transformed

this factor into two components: response time and input/output

operations (see section 5.2.1). Thus, to some extent, the examples

of the cases in Chapter 5 can differ from the original perspective

demonstrated in this chapter. While some modifications in Chapter 2

have been made to promote consistency, the case descriptions

below are mostly in their original form. As such, they also provide

a record of the writer's early thinking on the critical system

development problems and the key factors to monitor.

2.2 A PLANNING SYSTEM ^

A project was conducted between ETIP and the Public Buildings

Service (PBS) to explore the benefits of using life cycle costing

3 The following sections draw heavily from Lawrence and Garrity (1979).
in the planning and acquisition of Federal space. The project was

to develop a computer system to calculate these costs and institutionalize the process into the agency. ETIP money was obligated

to PBS to initiate the development. PBS later added much more money

of their own to the project as the requirements of the system expanded.

The project lasted over five years and resulted in a computer

model and procedures using the life cycle costing strategy. Several test cases were conducted with the system. Limited portions

of the model, particularly the economic analyses of specific space

configurations, were accepted by users. Other portions of the model

were not as successful due to data limitations. In addition, full

user acceptance of the system was not obtained by the end of the

project. This was partly due to their lack of control over system

development and to low level support by top agency managers.

2.2.1 Background and Objectives

Several years before PBS project began, the Federal government made some policy changes in the acquisition and management

of Federal facilities. The initiatives were primarily aimed at

improving these processes through the requirement of more systematic, explicit, and detailed planning. The initiatives were also

expected to increase innovative responses from contractors to Fed-
31

eral space needs and lower the overall costs of acquiring and operating office space.

The Public Buildings Service (PBS) of the General Services

Administration (GSA), an agency responsible for the design, building, leasing, appraisal, and maintenance of Federal buildings, became increasingly interested in life cycle costing (LCC) techniques

as a means to meet the requirements of the new directives. This

was particularly the case for the Office of Management and Budget

(0MB) Circular A-104 issued in 1972. A- 104 required PBS to conduct

cost analyses of property lease or purchase options. It also mandated analyses of costs over time, including those of operations,

maintenance, taxes, and insurance. While many of these considerations and analyses were already being conducted in PBS, the agency

wanted to develop a comprehensive LCC approach that could be computerized. This would facilitate evaluating and comparing costs of

alternatives.

Also, at about the same time, the ETIP program was developing

its interest in government procurement policies and practices.

ETIP was especially interested in how procurement policies and practices could stimulate technological irjiovation in the private firms

selling to the government. ETIP perceived that the PBS situation

might offer an opportunity to implement an LCC procurement and collect data on the effects. Accordingly, a project plan was developed
and agreed to by PBS and ETIP. The objectives of the project were

(ETIP, Note 1, p. 2):

• To develop and test, through actual experience,

the use of life cycle costing in the planning and

acquisition of Federal space.

• To stimulate, through Federal leadership, desirable building system innovations in the non-Federal

sectors of the economy.

• To develop a basis of future policy recommendations

regarding the use of life cycle costing in Federal

space planning and procurement.

While systems development was not an explicit objective in

the project, the idea of utilizing the model and accompanying procedures over the long term was in the minds of project designers.

Early versions of the project plan had statements such as "GSA/PBS

will use the system in their planning process" and "if successful

the experiment will provide justification ... to institutionalize

the use of life cycle costing in these processes (PBS, Note 2, p. 2).

Later, as the needs for and architecture of the system became

clearer, the project was expanded considerably to provide more resources directed towards system development (Mariscal and Company,

Note 3 )

2.2.2 Project Implementation

A model was developed and operational in PBS about three years

after starting. Its capabilities were much in line with the origi-
nal project objectives. The model consisted of three parts: a

space analysis component, a cost analysis component, and a cost

stream (LCC) analysis component. Theoretically, users could specify

functional requirements for space and then run the model to help

them further refine designs and develop cost figures. The analyses

were all easily altered in order to change assumptions or create

new alternatives.

Over a period of 18 months, the model was run about 40 times.

Users were more interested in the LCC estimates for specific space

configurations than they were in using the model as a design tool.

Even though the model's capabilities were not fully utilized, the

LCC figures were respected and accepted by users. Many of the operational runs produced figures later incorporated into PBS space acquis

tion proposals.

Actions were taken near the end of the project to build support for the model within PBS and with other agencies (Ostrander,

Note 4). The General Accounting Office conducted a review of the

model and supported its use in PBS. PBS provided money and the

two staff members to run the model. Users continued to make requests

for LCC estimates. ETIP's role eventually diminished until the

model essentially became wholly supported by PBS itself.
2.2.3 Important Factors in Systems Development

The LCC modeling project was an ambitious undertaking. While

the model was building on many existing procedures, it was adding

computerization to work normally done by hand and capabilities

previously difficult or impossible to develop. Results from model

use were expected to influence the agency’s planning and acquisition

processes and possibly affect the building technologies bid on specific acquisitions by private contractors. The model was supposed

to be used in three test acquisitions, with an evaluation following

to compare actual versus forecast costs.

The development was successful to a degree. The model was

built and tested numerous times, as mentioned earlier. Users accepted some results from the model and included them in their proposals for space acquisitions.

Several factors stand out as having facilitated implementation.

The model, supporting procedures, and computer programs were developed closely with PBS staff and the PBS project leader. The project

leader had a very high interest in the model and its techniques.

He saw them as promoting better cost analyses and in the end possibly improving competition and innovation in the bidding on specific

building, leasing, or renovating contracts (Ostrander, Note 4).

The model would help meet the requirements of 0MB Circular A-104

as discussed earlier.
35

In addition, the system reduced the time needed to do cost

analyses on various designs from weeks to hours. The analyses were

more easily adjustable for changes in designs and costs and thus

allowed more rapid turnaround for estimates.

However, some portions of the project objectives were not achieved.

First, the system was not fully tested in three space acquisition

problems. It was originally expected that the model and its results

would be part of three acquisitions which could be followed long

enough to allow comparisons between planned and actual costs. It

had also been expected, as mentioned earlier, that the system would

be more fully utilized in the planning processes and possibly become

a routine part of them. Initially this had meant using the system

as a tool in generating specific design alternatives. Later it

was hoped that the system would influence the contracting, building,

and operating of specific sites.

These plans were not fulfilled. The system was used primarily

to generate LCC figures. Other costs were still being calculated

by hand. The capabilities to compare and alter designs within the

system, or to use it to monitor contractor costs, were not utilized.

Overall, the comprehensive integration of the system into PBS was

not achieved.

A number of factors contributed to the changes in course from

original expectations. First, while the system functioned within
an important part of the acquisition process in PBS, its role was

small relative to all the other events which were part of the process. Numerous political, economic, and design issues are part

of any space acquisition problem, and life cycle costs influence

only some of these. Initial costs were the primary consideration

in decisionmaking. Ties with the final contracting, construction,

and building operation stages were small or non-existent. Thus,

while the system dealt with a problem important to PBS, its role

and its potential to influence acquisitions were smaller than expected.

Another contributor was that the system was developed and operated within the Office of Special Studies, a staff office attached

to the Commissioners of PBS. Since this office was not normally

responsible for conducting the cost analyses, the system was essentially developed outside of the units of which it was to be a part.

While this separation did not have to be a problem, in this case

it may have hindered gaining commitment from the user units. Several reasons were behind this problem. First, the model had the

appearance of a black box to users (Ostrander, Note 4). It was

difficult to make comparisons between the normal manual procedures

and the newer machine programs. Second, users were mostly concerned

about initial costs in a proposal, as this reflected common agency

practice. Since the system considered a range of costs in addition

to initial cost, users had to shift to an entirely new approach.

Training for users was needed, but it had not been a part of develop-
37

ment (Ostrander, Note 4).

Contributing to skepticism about the utility of the system

was that data for the model were hard or impossible to obtain.

PBS had committed itself early on to provide its own operations

and maintenance data (Galuardi, Note 5). However, requirements

of the model exceeded the capabilities of current data bases. The

model utilized a new structure for classifying major building subsystems and data were not yet commonly aggregated in this manner

(Penn, Note 6). Thus, it was difficult to validate the model by

comparing its analyses to actual cost figures (Ostrander, Note 4).

Finally, the model was not fully supported by top management,

the Commissions of PBS (Ostrander, Note 4). This was partly due

to uncertainties over what acquisition policies they were to implement. The model had been approved by the GAO in its investigation

and it appeared commensurate with the requirements of 0MB Circular

A- 104. At the end of the project, however, another policy, 0MB

Circular A- 109, was being promulgated for application to large scale

systems acquisitions. The role of A- 109 in PBS was unclear. There

also was uncertainty over the match between the A-109 requirements

and the LCC system.

Commissioner support was further ’’inhibited by normal management reaction to the increased specificity, logical consistency
and accountability implied in new applications of computer technology

to decision procedures" (Lawrence and Garrity, 1979, p. 17).

2.3 A DISTRIBUTION SYSTEM

For several years ETIP conducted a project with state and local

governments to develop and test the effect of procurement techniques

on industry suppliers. The project included objectives for developing

an information system that would serve both the procurement community

and ETIP. It would develop and distribute information needed by state

and local officials and provide a channel for ETIP to have influence

on techniques used. ETIP contracted the system development with two

nationwide procurement organizations having access to state and local

agencies. Several information packages were produced by a pilot system

and used by a number of agencies. Development was very slow and difficult however and the needs of agencies were found to be much broader

than expected. A system to meet the original project objectives could

not be developed.

2.3.1 Background and Objectives

As part of its research on government procurement and technological change, ETIP became interested in studying procurement policies

at state and local agencies. There was considerable interest in the

procurement community to initiate the ETIP research in this area.

Federal, state, and local procurement officials saw that increased
39

coordination and cooperation among all levels of government would

upgrade procurement techniques and performance for everyone (Sampson,

Note 7; Timbers, Note 8; Holding, Note 9).

While the basic ideas to be tested in the project were the

same as those in previous Federal level research, the state and

local environment presented some new problems for project designers.

ETIP had very little direct contact or experience at the state and

local levels. It would be difficult to establish this quickly with

a large number of agencies, especially considering the limited ETIP

budget and manpower. In addition, state and local agencies used

a variety of techniques and had a variety of problems. The market

was seen as very fragmented and this was a major barrier to developing the market force ETIP wanted in order to test its hypotheses

(ETIP, Note 10, pp. 1-3).

Several strategies were used to overcome these problems in project

design. First, two projects were developed, one at the state level

and one at the local (county, city, local agency) level. ETIP used

the advice and services of two nationwide professional procurement

organizations to help in the developments. These were the National

Association of State Procurement Officials (NASPO) and the National

Institute of Government Purchasing (NIGP). Both had extensive contact and experience with the procurement officials ETIP wanted to

reach.
40

ETIP then contracted the projects to NASPO and NIGP. Each project

had three main lines of work (ETIP, Note 10; ETIP, Note 11). One was

to develop a system that could identify procurement problems, develop

information to solve the problems, facilitate collaboration across

agencies, and generally improve procurement practices. ETIP viewed

the system as the key to gaining access to agencies and building

the coordinated market it needed. The system would both enable

work to be done and build support for the project at the agency

level

A second line of work was to conduct experimental procurements

testing the technological change hypotheses. These experiments were

to be part of the system development and one activity the system would

support in the long term.

The final line of work was to develop a means of supporting the

system in the long term. All parties envisioned that the system would

continue functioning after project resources were expended (Cornett,

Note 12; ETIP, Note 11, p. 10; ETIP, Note 10, pp. 6 - 7 ).

2 . 3.2 NASPO Implementation

NASPO began its work by conducting a survey of state procurement agencies in order to identify products which involved procurement problems the state needed to solve. A long list was pro-
41

duced and then refined down to seven common products, including,

for example, air conditioners. Using volunteers from interested

states, NASPO formed task forces to develop the techniques, product information, criteria, tests, or other items needed for procureing the products.

Developing the packages proved to be more difficult than expected

however. The needs of each state were often too different for

the development of a consensus package (Cornett, Note 13, pp. 21-

24). In some cases the items needed in the package were not readily

available from other sources as originally expected. Since the task

forces were dependent on the voluntary efforts of busy state officials,

it was difficult for them to generate new information on their own.

Eventually four product packages were produced and sent to states

for use in their next procurement cycle. A significant number of buys

occurred with the packages, although in most cases the packages needed

further onsite modifications before they were used (Wagner and

Zeldis, Note 14). The major conclusion to be drawn from the package

developments was that each agency had its own preferred procedures

and that the differences among them were fairly wide. Aggregating

the market by standardizing procurement methods and timing was net

very likely.

2 . 3.3 NIGP Implementation

The NIGP project began several months after NASPO and followed

the same approach. For about a year, procurement problems were solicit-
42

ed from NIGP members. Sixteen common products were identified among

the list of these problems. NIGP then began to develop procurement

packages for the sixteen products. Like NASPO, NIGP did not start out

with the internal technical capability to develop the packages. However, instead of the field task force approach used in NASPO, NIGP chose

to hire a coordinator of their own and work closely with individual

officials in the field. This process had the same problems as experienced at the state level. It was slow and difficult to sustain constituency interest in the packages or the products. Only four product

efforts survived and packages were not completed for any of them.

2 . 3 .^ Project Reviews and Revisions

Over a period of time within the first year of both projects,

it became apparent to ETIP and others that some of the barriers

identified in the project plans were more significant than anticipated.

Development of the procurement packages had proven to be very time

consuming and difficult. New information had to be developed, packages

had to be specifically designed for individual units, and in many

cases other state and local officials besides those in procurement

had to be involved in order to switch to the new methods (Cornett

Note 13 , pp. 23 , 42-44). This ;vas a more active role than originally

planned. In addition, too much attention had to be devoted to the

packages over system development and financing (Berke, Note 15;

Berke, Note I 6 )

Eventually ETIP decided to change the structure of the project

and assert more control (Berke, Note 17; Berke, Note I 8 ). Package
43

development would continue only for industry products of interest

both state and local agencies. The most innovative agencies would

be selected instead of trying to reach everyone. ETIP would try

to find more technical assistance in other Federal and private

organizations. Above all, ETIP would push much harder for experimental buys.

While these changes were not significantly different from previous strategies, they did bring out a philosophical split in the

project team. Originally everyone had agreed that a system was

needed both to facilitate change in procurement practices and implement experimental buys. Development of the system was to be based

on early studies of state and local problems and a small number of

experimental purchases. Successful, innovative procurements would

increase the attractiveness of change to other agencies and thus

promote more experiments and more market aggregation. This in turn

would lead to a larger, popularly supported system which would remain

beyond the duration of the ETIP projects.

NASPO now believed that the problems they were encountering with

package development were indicative of a considerably larger resistance

to change. Pushing experimental buys using different techniques was

probably premature in most cases. A wider scope of change in agencies

was needed before experiments would be feasible (Cornett, Note 19).

NASPO felt that a better approach would involve more emphasis

on developing a system that could support slow, incremental changes
in procurement when agencies decided to make them. The project

should build up contact and trust with state and local officials

and market the system capabilities. Experiments would evolve out

of influencing the design of agency initiated changes.

NIGP agreed with the difficulty of making changes in state and

local procurement, but they continued to push the package development

- experimental buy approach to change (Spangler, Note 20; NIGP, Note

21, pp. 4-5). NIGP staff felt that new packages were in demand and

only needed to be more specifically designed and marketed to agency

users (Cornett, Note 19; O'Connor, Note 22; Arnold, Note 23). This

would take more time than originally expected and thus a viable

program of experiments would also require more time to evolve.

ETIP took a harder stance on the need to have procurement experiments. This was based partially on the belief that experimentation

was the best approach for learning about the state and local

environment and on the desire to have some specific results (procurement changes) in the project (Lewis, Note 24; Lewis, Note 25).

This emphasis prevailed and subsequent work became almost totally

oriented around products, innovative agencies, procurement packages,

and schedules for experimental buys. The philosophical differences

remained, however, and working relationships became somewhat strained

(Wagner and Zeldis, Note 14, pp . 111-1 - 111-4).
Even with an increased emphasis on packages, experimental buys,

and NIGP-NASPO coordination, the remaining months of the second

year were no more productive than those of the first. This, along

with the fact that both NIGP and NASPO contracts would soon terminate, precipitated an entirely new internal review of the projects

at the end of the second year. Work on product packages was continued, but a significant amount of ETIP, NASPO, and NIGP attention

was focused on producing reports of accomplishments and projecting

future payoffs for ETIP managers. Work on the systems aspects of

the project was very minimal during this period.

ETIP finally decided to end its involvement at the state and

local level. The barriers to experimentation were considered larger

than ETIP had originally expected or wanted (Lewis, Note 26). The

state agency project was terminated shortly thereafter when the

contract ran out. The local agency project was extended at no cost

for another year since a few of the package developments appeared

likely to result in experiments (Lewis, Note 27). Not much progress

was made during this extension, however, (Hulick, Mote 28).

2 . 3.5 Important Factors in Systems Development

The state and local procurement projects were some of the larger

undertakings in the ETIP program. The project was to develop a system

to identify procurement problems, set up a means to develop and transfer

information between agencies to help solve these problems, and promote
the use of improved techniques with potentially hundreds or thousands

of individual agencies. In addition, this system was to facilitate

coordination between agencies in order to develop an aggregated market.

This was to permit a powerful test of the ETIP hypothesis that government procurement could act as an incentive for industrial suppliers

to improve the technology embodied in their products.

Adding momentum to the project was the universal agreement among

project participants and the procurement community that a system was

really needed. Both the top administrators of ESS, ETIP's Federal procurement partner, and GSA, its parent organization, expressed great

interest in the potential to exchange information routinely with state

and local agencies. Both saw a long term Federal commitment to such

a system. NASPO and NIGP each thought that a system was needed to work

with the procurement problems of their constituencies and would increase

their organization's influence on policies and practices (Hall, Note

29 ; ETIP, Note 30 ). Other procurement and industry officials expressed a need for a system to promote information exchange across

agencies (ETIP, Note 31 ).

In addition, the items needed by agencies seemed specific enough

to be producible within the modest resources envisioned for the project.

It was thought that many of the items already existed in some of the

more innovative Federal, state, or local agencies. Meeting the needs

of individual agencies would simply be a matter of identifying the specific information problem, an information source, and exchanging mater i-
47

als. Little modification of existing procurement items would be needed.

Thus it would appear that several factors important in developing

and obtaining commitment to a system were present in early project

stages. First, there were real problems in state and local agencies

and a system appeared to be a good way to handle them. The concepts

of the system were supported by a wide range of officials representing

Federal, state, and local procurement agencies. A long range perspective was taken early and objectives for assuring post-project support

were explicitly included in both state and local project plans.

Second, by working with organizations made up of people the project wanted to influence, ETIP was able to integrate user concerns into

project design, achieve early credibility with users, and obtain a natural route to keep them involved in system development. With NASPO

and NIGP having large roles in systems development, it appeared that

the system was being built in the field by users for users.

Later progress with system development did not meet the early

expectations however. As described earlier, significant difficulties

arose in trying to produce information to meet agency problems. Building support for the system was hampered by this lack of success and

eventually the system aspects in the project were downplayed in order

to concentrate resources on specific procurements and agencies. After

three years of work, ETIP terminated the project, unable to develop

a system that could meet the planned technical and research objectives.
48

The lack of success clearly demonstrated how important problem

identification can be to a system development. Project planners had

thought that a simple system to exchange information packages would

help solve procurement problems and provide a channel for influencing

agency practices. This thinking was too narrow. Agency practices were

found to be fairly unique and experience in one agency was not always

readily transferable to another. Procurement officials often needed

well tailored technical guidance, new facilities and/or equipment in

addition to new information.

Achieving a long term commitment from users to support the system

also proved to be more complex than expected. The original idea for

institutionalization was that agency users would gradually increase

their support to operate the system over the life of the project. At

the end of project, ETIP would no longer provide any funding. It was

found however that procurement officials were not the only ones who

might have to be involved in making a commitment of resources. Many

different officials in state and local government could have a role

in procurement policy and practice. For example, legislators controlled

procurement through laws and appropriations. Their direct knowledge

of procurement could be small or nonexistent and yet they would be key

supporters for a system serving their respective procurement officials

(Spangler, Note 20).

The reality of the procurement environment presented a critical

problem to the project team. The system development had been conceptual-
49

ized, designed, and funded around one small facet of the problem. A

much expanded scope was needed and it exceeded the resources available

in the project. Funding had been set early in the project plan phase

and no flexibility was built in to accommodate major shifts. The technical expertise needed to work closely with agencies was not readily

available.

There were also significant implications for institutionalization of the system. With different groups involved in procurement,

some direct users and others only supporters, gaining commitment

to the system in the long run would involve an expanded number of

decisionmakers. The influence NASPO had outside of the procurement

community was much weaker and the evidence needed for gaining support

unknown. The project was not really organized for a coordinated

drive to find and influence the key officials in each case.

Some changes were made in project design to compensate for these

problems. New staff were hired into NASPO and NIGP. Attempts were

made to establish ties with government and private organizations

possessing technical capability in procurement. The number and

types of procurement problems to be handled were reduced and only

innovative agencies were selected for participation. Finally, NASPO

turned to industry representatives to gain more support for the

system.
50

Unfortunately, these changes did not alter the basic outcome of

the project. It was too difficult to compensate midway into the develop'

ment for the lack of knowledge at planning stages. The flexibility

needed to meet the broadening problems was not built into the development at the start.

It should be noted however that a firm case was not made for or

against developing a system. The problems at the state and local level

were real and things could be done that were not being done. A new

system to handle these problems might have been designed given the proper timing and resources. This system would have probably been much

different than the one planned in the ETIP project, and it is likely

that developing it would have exceeded the capability and interest of

the parties involved.

2.4 A DATA BASE SYSTEM ^

ETIP discussions with the Small Business Administration (SBA)

revealed that the agency was concerned about the small share of

government research and development (R&D) work contracted to small

businesses. An initial study of the problem showed that SBA field

The title of the system may be somewhat misleading. The system

started out more as an evaluation process than a data base storage

center. However, the data base aspects became increasingly important

during the project. The system title was chosen to reflect this

changed emphasis.
51

officials needed help justifying R&D set-asides to Federal agencies.

ETIP and SBA then agreed to conduct a project that would develop

a system field officials could use to improve the set-aside process.

A system was designed to match the needs of an agency procurement with the technical experiences or capabilities of firms. This

would help generate a list of technically competent sources that

could be used to justify a set-aside. Two versions of the system

were created, one manual and one computerized. Each had some success in identifying competent firms. The computer version was eventually more successful since it combined speed with a data base

of sources users also needed. Outputs of the computer system were

used in some real set-aside cases. It did not, however, gain wide

acceptance among users since it was expensive to operate. The project ended without any long term commitments to either the computer

or manual versions.

2.4.1 Background and Objectives

In 1974 ETIP was discussing several potential projects with the

Small Business Administration (SBA). ETIP was interested in small businesses because it was widely claimed that they were major sources of

technological innovations in the U.S. There was concern that these

sources were being underutilized by Federal agencies. ETIP thus undertook an effort to learn how SBA policies were affecting small firms

(ETIP, Note 32).
52

Discussions between ETIP and the central SBA office led to ideas

for increasing the share of research and development funds (R&D) to

small businesses. The central office felt that small businesses were

not getting enough of the government's R&D work (Charney, Note 33)*

This was partly because (Innovative Systems Research (ISR), Note 3^,

p. A) :

• Proposal preparation for a small firm would be relatively

more costly for them than for a large firm.

• Contracting with unknown small businesses was risky com- pared to contracting with the known larger firms.

• It was difficult to determine when a R&D project should

be set-aside for small businesses.

It was decided that this last area, set-asides, was one where

SBA interests matched those of ETIP. SBA, through set-asides and the

related certificates-of-competency , influenced the flow of funds to small

businesses. A set-aside action could be taken by a government agency

to allocate part or all of the work in a contract to competitive bidding

by only small businesses. A certificate-of-competence could be issued

by SBA to document a firm's capability, primarily in financial matters,

to perform the work in a contract to be awarded the firm. These actions

provided ETIP a limited sector in which to investigate the assertion

that small business capabilities were being underutilized. Specific

government policies could be identified, agency actions under them traced,

and resulting impacts on small businesses studied within specific setaside cases.
53

The major problem with R&D set-asides was that SBA field representatives were having a difficult time identifying R&D work that should

be set-aside and then making the case to the procuring agency (ISR,

Note 35, pp. 1-4 - 1-6). SBA representatives, known as procurement

center representatives (PCR’s), had to maintain liaison with agencies

procuring R&D services and learn what would be solicited and when.

The PCR then had 15 working days to justify a set-aside to the agency

contracting officer. This justification had to show that there were

small businesses capable of performing the work. Finding these businesses, or sources, and evaluating their technical competency was difficult, especially with the uncertainties of R&D work.

It was thought by ETIP and SBA that some system could be developed

to help the PCR find sources and make set-aside cases to the agency.

This system might also assist in certificate-of-competency cases since

data common to both actions might be stored and easily accessed. A

project was developed around this theme and the objective was to develop and test a means for regularly and routinely determining the competency of small R&D firms to perform contract R&D for the Federal government (ETIP, Note 32, p. 1). SBA was to conduct the project with the

support of a contractor. There were to be six main tasks:

1. Review set-aside and certif icate-of-competency cases to

learn how they are done and the problems involved.

2. Develop a preliminary plan for the operation of a system

to support the set-aside and certificate actions.

3. Develop an operational plan including the specifics of

system design.
54

4. Plan tests of the system.

5. Conduct tests and refine the system.

6 . Operate the system with real cases for 12 months and

evaluate its performance.

The project was scheduled to last 22 months from the date of award to

the contractor.

2.4.2 Project Implementation

Work began about one year after the plan was completed. A small

firm was hired by SBA to perform the tasks and they kept to the original schedule fairly closely. Within half a year, the project was in the

operational planning stage. The system, called the Preliminary Determination of Qualifications (PDQ) system, was designed and ready to be

tested in the field.

Briefly, the system worked as follows. Field representatives

first were to identify pending agency R&D contracts and obtain the statements of work. A procedure had been designed for PCR's to then classify

the statement of work in a manner which would identify the major items

involved, such as actions and products. Once this set of descriptive

items was identified, PCR's were to place this information on a questionnaire to be sent to potential small business sources. The officials

also had the option of calling the firms. Small businesses receiving

the form were to classify their experiences or capabilities similar

to the structure for the work statement so that their experience could
55

be compared to agency needs. SBA officials were to receive this information, determine if there was a match, and finally develop a list

of potential sources. The lists were to be used in set-aside justifications made to agency contracting officers.

The system was presented to the Associate Administrator of SBA

to gain approval of the field work. With this approval, the contractor

then held the first field workshop in Philadelphia with SBA staff from

several regions. The workshops were designed to train the staff in

the PDQ system and promote testing on set-asides contracts already completed. This would allow the project team to observe how well the various procedures, forms, and concepts of the system worked under nearly

actual conditions.

The workshop went well and several SBA personnel became involved

in trying the system on additional retrospective cases (ISR, Note 36,

p. 4-22). Eight cases were tested with the system and the results were

good. In most cases, the firms that had been selected in the actual

procurement were identified as technically competent by the PDQ system.

This result dispelled the fear by PCR's that the system would be inaccurate. However, participation in the testing was smaller than expected

and a number of implementation problems surfaced (Braudy and Albert,

Note 37). PCR’s claimed that the system did not really help them with

two major problems: that of finding sources to begin with and assuring

bids from competent firms once a set-aside was made. Also, it seemed

that the firm data needed by the system might not be readily available
56

within the 15 day limit the PCR had to make a case. The contractor

felt that the first two problems were not solvable in the system and

that more data was needed on the time limit issue before a change could

be designed (ISR, Note 36, pp. 3-1 - 3-2; Braudy and Albert, Note 37).

It was here that the idea of establishing a computerized system first

arose.

Since the system had generally been accurate in the tests and

did not require any major changes, it was decided to move into the

operational phase of development. The purpose of this phase was to

run the system for one year using real cases underway in the field.

PCR's were to be trained to use the system and then given the materials

to apply it to new solicitations. The contractor was to monitor the

applications, verify results, and evaluate system performance.

Two different lines of work evolved in this phase. First,

the contractor and the SBA central office began pushing for real

time applications of the system. Two more regional workshops were

conducted in the western United States to introduce the system and

train personnel. The receptions at these sessions were not as favorable as those in the east, however. Opposition to using the system

in actual set-aside situations arose, primarily because it seemed

to SBA personnel that the system overlapped their "turf" and conflicted with their expertise role in the set-aside process (ISR,

Note 36 , p. 21; Penn, Note 38)* Attention was returned to the east

where implementations appeared more likely.
57

On returning to eastern regions, it was decided that more emphasis

on retrospective cases would be made. It was expected that this would

be easier to accomplish and would build a track record that could in

turn make it easier to get real time applications. A few retrospective

cases were obtained in this switch, but they were very slow coming in.

The poor response prompted the SBA central office to send a letter to

the non-participating regions. In the end only 10 cases were obtained

in the operational phase. This was disappointing to the project team,

but the PDQ system nevertheless maintained its accuracy record and good

standing.

The limited success with real time cases was compensated somewhat

by another development which opened up a whole new line of work. Several months after the western workshops occurred, the SBA central office

in Washington began discussions with one agency in the Denver region

over a potential system application. The agency had a large number

of procurements to make in order to obtain several environmental impact statements. They had a short amount of time to get the contracts

out and were open to considering small business set-asides (ISR, Note

39, p. 3-D* There was a good chance that this situation would provide

the project with the real time applications it needed.

A number of retrospective cases were obtained from the agency

in order to redesign the PDQ system for impact statements instead of

R&D contracts. Four cases were examined and then tested in a revised
58

PDQ approach. Results were promising in that the system produced technical competency ratings in line with the judgements of the agency.

The problem was that the PDQ process took time which most PCR's could

not provide.

To overcome this problem, the contractor proposed, and SBA accepted, the idea of trying to computerize the PDQ. A data bank of small

businesses with the potential capability to perform agency work would

also be developed. PGR staff within the agency would then take a specific solicitation, analyze its requirements, and quickly search the data

bank for sources. The resulting source list would establish a case

for a small business set-aside, hopefully well within the 15 day limit

given to PCR's.

Given the press of the agency business, project focus shifted

almost exclusively to the computer system. A number of new obstacles

had to be overcome. First, the contractor found that it was impossible

to classify impact statement work in a general way that would allow

each firm to be rated as capable of producing a class of statements.

Each procurement for an impact statement was unique, and had to be compared with specific experiences in the firm. Thus the system would

have to search each firm in detail for each procurement rather than

turning out a subset of firms considered generally qualified.

Another problem involved the actual comparisons of procurement

requirements and firm experiences. This was to be done by the computer
59

and it, therefore, required an elaborate scheme for matching words.

For example, a firm's previous experience in producing an impact statement had to be judged as the same, similar, or unlike the requirements

needed in the new procurement. Only word by word comparisons were possible, and the context of phrases of sentences was lost.

Nevertheless within about a month, the contractor had an operating

computer version of the PDQ system. A data bank of 125 firms who could

perform impact statement work was developed. A number of demonstrations

were then held in the agency. In one test case, a completed set-aside

procurement was used. The qualified firm resulting from system processing was the same one that had actually been awarded the contract.

This was a great success for the project and established support for

the system in the agency (Braudy and Albert, Note 37). Subsequently,

the system was used in five real time cases for the agency and five

for other SBA locations. The information generated by the system was

used in decisions to make set-asides and to establish a source list

for sending solicitations (ISR, Note 39, p. 3-24).

The project came to a close several months after the computer

system was developed. A final report and system documents were delivered to the SBA central office (ISR, Notes 40, 4l , 42). It was concluded by the contractor and SBA that the computer PDQ was the only

viable approach to generating sources and facilitating the set-aside

process (ISR, Note 39, pp. 4-4 - 4-7). The manual system worked, but
60

was slow and unattractive to PCR's since it required too much of their

time to analyze documents.

The computer system was not supported beyond the project’s end,

however. Even though the Denver region and the test agency were both

interested in using the system further, they did not have the resources

to support it. The computer version was expensive to run since there

had to be long searches through all the firms (Penn, Note 43). In addition, PCR’s outside the Denver area generally didn’t see the need

for the system or thought it was too complex to implement (Charney,

Note 33). These circumstances didn’t generate much momentum to continue

supporting the system and the SBA central office decided not to do so.

Although no further plans were made to implement the system at the time,

some staff at SBA thought the system might resurface years later when

another national source system was completed (Charney, Note 44). This

system was to be similar in design and function to the computerized

PDQ, but was to cover a broader range of contract work (SBA, Note 45).

Thus the concept of a PDQ system was, in the end, further supported

by SBA.

2.4.3 Important Factors in System Development

The PDQ system development with SBA nearly achieved the objective set out in the project plan. A means for evaluating the technical competency of small R&D firms was developed on schedule and

tested in the field with ultimate users. Tests results were promising
61

and under certain conditions the computerized PDQ system seemed to meet

a real need.

On the other hand, the system did not achieve the popularity and

support desired by ETIP and SBA. Portions of development which depended

on close contact and cooperation with field personnel, the end users

of the system, were not especially successful. The PDQ system was not

generally seen by users to be a regular and routine option they could

use to meet their set-aside objectives. In the end, the SBA decided

not to institutionalize the system.

The mixed success with the PDQ development seems to have been

partly a function of problem focus. Initially, the project team conceptualized the system around the difficult problem of matching small business firms to the requirements of a specific solicitation. The system

was to help establish the case for the PCR that technically competent

firms existed to do the work. Even though the resulting PDQ

manual system had some modest success with the problem, it did not

gain wide support among PCR’s. The procedures were seen as too time

consuming for already busy officials (ISR, Note 39, p. 3-31).

More importantly, PCR's, the system users, viewed technical evaluation as their key function in the set-aside process. The system not

only overlapped their "turf”, but tried to reduce their complex function

to simple, standard procedures. It was difficult for them to accept and

use the system as a substitute for their skills.
62

The resistance was moderated in the one case, however, where the

development shifted to the problem of finding sources. The agency involved needed to procure several contracts quickly, but it did not have

the source list to start contacting firms. The PDQ process was computerized in this case along with a data bank of sources. This system was

able to find potential sources rapidly and it gained support from the

agency and the PCR (Charney, Note 33). Unfortunately, it could not

be expanded to cover other agencies and gain wider support. The system

was expensive to run, especially considering the narrow field of setasides it covered, and the project at this point did not have the resources to build more.

The importance of finding sources might have been noted sooner

in the project had backgrounding been more extensive before project

design. A more thorough study of the environment in which the system

was to be embedded might have demonstrated the need for a system to

find sources. It might have at least provided more guidance on how

to enhance the critical matching and evaluation functions which users

felt were their main contribution to set-aside actions.

Involving the users earlier in the development probably would

have compensated some for the lack of backgrounding. As it was, the

field workshops for introducing the system to users were the first

chance users had to see and react to it (Charney, Note 33). It was in

these sessions that the contractor learned how great the need was for

a system that would find sources (Braudy and King, Note 37). By this
63

time though, the system design was fairly advanced and a changeover to

the source problem was difficult. It was also a critical period in

which to be uncovering basic resistance. Further development according

to the original strategy depended on gaining user support and confidence so that both retrospective and real time cases would be made

available to test the system. In the end, the quest for cases was not

that successful and a solid track record could not be established.

Extrapolating from this example, one might conclude that the

project strategy in general was ill-suited to the desired objectives.

The original strategy included developing a system in the central

office around the problem of matching firms and solicitations.

Test cases and data were to be obtained from the field as needed.

The system was then to be tested in the field with users by applying

it to real time cases. While this wasn’t a bad strategy by itself,

a number of field characteristics made implementing it difficult.

For example, the relationship between the central office and the

regions was not that strong (Charney, Note 33). This made it very

difficult to obtain the field cases needed to design, test, and

validate the system. The situation was further complicated by the

pressure PCR's were under. Generally there weren't enough of them

available for every agency to have one in residence. PCR's often

had to maintain a number of liaisons with different agencies and

this didn't give them much time to act in each case. Finally, R&D

contracting was only a small portion of the total of all contracting. While these characteristics had provided the justification
64

for the development in the first place, the resulting manual system

unfortunately did not really help PCR's. The project might have

benefited from more early and extensive contact with the field conditions and thus have avoided the problem of investing in one design

too early.

It also must be noted that the strategy lacked plans for implementation. Tasks were neither set out in the beginning nor added later

which would have guided resources towards developing long term support

for the system (Penn, Note 44). While it is unlikely that the manual

PDQ would have gained this support, the computer version was attractive

to the one agency where it had been developed. More resources might

have allowed the project to gain the track record originally desired

with the manual system. This might have then generated the follow-on

applications needed and the user acceptance desired.

Even though the PDQ system was not institutionalized, its concepts

were very much supported within the SBA central office when the project

was over (Charney, Note 44; SBA, Note 45). A new system similar to

the PDQ was to be created which would apply to many kinds of set-asides

and contain a very large data base of firms to scan with a computer.

If anything, the PDQ project demonstrated that such a system could be

constructed, that it would have to be computerized, and that it would

have to be matched more closely to the environment in which it was to

be used.
65

2.5 SUMMARY AND COMMENTS

Three example system development cases from ETIP have been reviewed in this chapter. The purpose of the cases was to identify factors in development which are important to project success. Each case

included a history of the project and a section on important development

factors

A number of different factors have been identified. For example, one major factor that influenced progress in all three cases

was the need to know the user environment. Developers conceptualized and committed themselves to a system design based primarily

on their initial understanding of the environment. Later exploration showed that the environment was more complex than originally

thought. In all cases, the original information problems were often

related to many other internal agency problems, with the latter

being of higher priority to users. Developers found that it was

difficult or unwise to isolate the project from other agency problems, and equally difficult to deal with them under the allocated

resources. In one case, developers also found that the agency worked

much differently than expected (i.e., the PBS planning system).

Even though parts of the PBS system were successfully developed,

the system could not be tied to the agency processes developers

originally planned to influence. Its impact on the agency was thus

greatly reduced.
66

Another common development problem was that developers did not

discover key problems, or realize their significance, until well into

the projects. Generally, it was too late to compensate at these points

by redesigning the system or the project. Significant portions of

the allocated resources were already expended or committed, leaving

little flexibility. From a management perspective, the projects might

have benefited from earlier, intensive reviews of strategy and objectives and more routine monitoring of progress. Resources might then

have been redirected in time to fit the changing circumstances.

Factors like these from each of the cases have been used to develop a framework of important factors for ETIP managers. Other systems literature has also been used and this material is presented

in the next chapter. Chapter 5 presents the resulting framework.

Factors in the framework are supported by references to the three

ETIP cases just discussed.
67

CHAPTER 3

SYSTEM DEVELOPMENT PERSPECTIVES AND

PROBLEMS IN THE LITERATURE

3.1 INTRODUCTION

The purpose of this chapter is to review the perspectives and

problems for system development that are found in the various sectors

of the systems literature. The underlying goal of this work has been

to conduct a broad review of the literature that would capture the

diverse views offered in the many different kinds of sources. Some

literature sectors have not been adequately examined, such as the

defense system literature and the materials from systems firms in

the private sector. In addition, no attempt has been made to review

in depth each literature area. This task was beyond the needs and

resources of the research project. It is hoped that the following

descriptions give the reader a "feel" for the literature available

and establishes some of the background which helped develop and support

the dissertation research.

The following sections both review the literature and identify

key problems and research needs. First, an initial section briefly

overviews the system development literature. Some of the different

sectors of the literature are identified. Next, several sections

describe and illustrate in more detail the materials available in

the different literature sectors. Examples of sources have been se-
68

lected that are referenced elsewhere in the dissertation. Finally,

several sections in the second half of the chapter identify and review

the key problems and research needs found in the systems literature.

These issues have been a central part of the development of the dissertation and readers may want to review them in order to gain the

proper perspective for succeeding chapters.

References are made in Chapter 3 to the objectives and approaches of the dissertation. These have been included in order to link

the dissertation research more closely with the needs found in the

literature. Readers may also find it valuable to review these sections (particularly the key problems and research needs) to understand the research design in Chapter 4 and the resulting monitoring

framework in Chapter 5.

3.2 SYSTEM DEVELOPMENT LITERATURE

3.2.1 The Systems Literature in General

There is a large and varied amount of written material available on systems and their development. Discussions can be found in

many different types of literature from both public and private sector

sources and covering both hardware and software applications. For

example, system development is discussed in literature areas devoted

to management science/operations research (Ackoff, 1967; Boland, 1978)

adoption of irjiovations in organizations (Air Force Systems Command,
6Q

1966; Dietrich, 1977; Patterson, 1977), and the acquisition of systems in government agencies (Comptroller General, 1976; Office of

Federal Procurement Policy, 1976). There has especially been a long

term interest in systems in the defense related systems acquisition

literature (Air Force Systems Command, 1966; Hill, 1970; Peck and

Scherer, 1962; Peterson, 1980).

Interest in systems is increasing in many areas (Chestnut, 1967,

pp. 361-378) and the literature is expanding rapidly. This is particularly the case for information systems, where much more attention

is being devoted to their development and use. Several forces behind

this expanding area are:

• The success in automating many routine clerical activities

in the early applications of management information systems (Chervany, 1978; Gorry and Morton, 1971; Karger and

Murdick, 1977).

• Technological advances in hardware and software which can

be used to process and store information (Chervany, 1978;

Edelman, 1977; Mumford, Land, and Hagwood, 1978).

• Increasing interest in decisionmaking tools which depend

on information systems, including the emerging interest

in decision support systems (Alter, 1977; Arnovick and

Gee, 1978; Karger and Murdick, 1977; Keen and Morton,

1978; Vazsonyi, 1978).

• Increasing numbers of factors needing consideration in planning and executing public and private programs (Welsh and

Lee, 1979).

• The need for more data obtainable only by routine long range

monitoring, such as needed in program evaluations (Comptroller General, 1978; Kraemer and King, 1978; Lorenzen and

Braskamp, 1978; Miller and Wilier, 1977).
70

A number of different professional and academic areas have arisen

or taken interest in information systems and have become important

centers for development methods and experiences. For example, Keen

and Morton (1978, p. 79) identify six professional groups from which

have evolved different perspectives on information systems: computer

science, management science, behavioral science, data processing professionals, management, and decision support.

Many different problems and factors in system development can

be found in these areas. Some of the more common factors are the

need for user involvement in a design project, identifying the information needs of managers, tailoring system outputs to different decisionmaking levels, and the need for top level management support

in system development. Since system developments can be large and

complex undertakings, a large number of factors like these can potentially be applicable in any one specific situation. This has led

to elaborate reviews or models of development which attempt to cover

the many important factors in detail. For example, Ein-Dor and Segev

(1978) review 120 organizational variables they have found to be related to information system implementation. Taggart and Tharp (Note 1)

present 50 different models of initiating information system developments. A developer reviewing the literature for important factors

will find more than enough to work with.

Relevant literature for information system developments is not

limited to these several specialized groups however. Literature pro-
71

duced in other sectors for other kinds of systems can also be very

useful for information system projects. First, many of the development problems are very similar. For example, the need for top level

management support is a fairly common factor found across a diverse

number of sources. Slightly different perspectives or applications

of this factor can be valuable in gaining an understanding of its

importance. Second, there can be differences in the kinds of factors

considered between sectors. For example, project management issues

are more commonly discussed in defense related literature than in

behavioral science literature. Concentrating on one sector may thus

provide a narrower perspective on the relevant factors than needed.

It is beneficial then to take a broader approach to reviewing

the literature. There are several other ways to divide the literature which make it easier to describe and categorize.

3 . 2.2 Different Sectors of the Systems Literature

Literature relevant to systems development can be divided by

a number of different dimensions. Three which seem most generally

useful are:

• Factors considered.

Literature sources can be divided by the range of variables considered. For example, computer science literature

often concentrates on technical aspects of design while

behavioral science sources concentrate human impact or behavior problems (Keen and Morton, 1978, p. 3^).
• Type of systems.

This can include any type of hardware or software system:

weapon system, management information system, computer system, evaluation system, decision support system, etc.

• Major perspective on development.

Systems literature can be divided into case studies of development, factor studies on important aspects of development, models of system development, and descriptions of

specific designs or design issues (Ginzberg, 1978a, p. 57).

Developers needing guidance for specific information system projects,

or other kinds of system projects, will find a range of sources along

these dimensions. They will also find that it is useful to review

a broader range of sources than simply ones specifically related to

their respective application.

In order to discuss the literature further, it appears most useful to divide sources by the kind of perspective taken on development. Several reasons are behind this approach. First, the categories on this perspective dimension are fewer and broader than the

others. Second, these categories capture techniques and experiences

from the diverse sectors of the literature. Third, this approach

is the easiest way to generally categorize the literature.

The following sections review in more detail the systems literature relevant to development by describing and illustrating the different categories of perspective. The latter half of the chapter

then describes a number of problems for system development based on

these areas.
73

3.3 A REVIEW OF SPECIFIC LITERATURE AREAS

The following sections review in detail literature relevant to

system development. The literature is characterized using four different perspectives:

• Case studies of specific developments.

• Factor studies which examine limited numbers of variables

important in development.

• Models of system development.

• System design sources which describe specific systems or more general technical design problems.

Each perspective is generally described and then followed by specific examples which illustrate the kind of material available. It

should be noted also that some sources overlap these areas. In some

cases the same literature source may be used to illustrate two different perspectives. This characteristic is discussed further in section

3.4.2.

3 . 3.1 Case Studies of System Developments

Reports on actual system developments can offer the most detailed picture of what a development is actually like. One could

consider much of the literature as consisting of case reports, since

many authors use an actual experience to develop lessons, strategies, or specific designs. For the purposes of this section however,

a case study was considered to be a report which focused on the
74

details of how a development was actually conducted, including parts

on the information problem, day-to-day events, key stages and incidents, management and organization, and user interaction. Case

studies emphasizing other factors, especially on design, will be

discussed in later sections.

Even with the narrow focus just discussed, there is a considerable spectrum of case studies available, from extremely short to very

lengthy and from hardware to software. Many articles, particularly

in management science literature, leave out most of the events between

the initiation of a project and its final conclusion (e.g.. Fudge

and Lodish, 1977; Chen and McCallum, 1977). Others leave out the

perspectives of different groups involved, such as explaining a development from the manager's viewpoint and including little if any user

information (e.g., McGrath, 1970). Often times lessons are included

in a report which appear to go beyond the evidence available in the

report (e.g., Krasnican, 1971). These shortcomings are not serious

deficiencies, however, since the perspectives and lessons offered can

be quite valuable in gaining an understanding of what really happens.

Three examples of management information system developments

were abitrarily chosen to illustrate the cases available and their

variability in content and style. As pointed out earlier, developments from both hardware and software systems are relevant to the

concerns here. Any mix of these could have been selected for use

here. For example, an interesting example of a weapons system de-
75

velopment is provided by McNaugher (1980) who reviews the background

and influencing factors in the development and deployment of the Ml6

rifle. Mann and Williams (I960) also provide an excellent case history of the changeover to an electronic data processing system.

The first example below is a brief case study of an information

problem in a corporation and the system designed to solve it. The

second example is a description of a computer system development which

provides more detail about the case than commonly found in much of

the literature. The third example provides a description of a failure.

3. 3. 1.1 A Brief Case Study

On the briefer side there are reports exemplified by Fudge and

Lodish (1977) on a planning system. The authors were involved in

developing a system for the national sales force of an airline. The

problem behind the system was to help the sales staff allocate their

time among their clients such as to maximize sales and profits for

the airline. This particular report does not include much on how

the authors got involved in the development or how the problem was

defined, but refers to earlier articles which do not provide these

details either (Lodish, 1971, 1974).

The narrative starts at the point where a model for designing

a strategy has been developed and is about to be introduced to sales

personnel. A 1 1/2 day seminar led by corporate managers was used
to acquaint sales staff with the model. Managers and salesmen spent

much of the time discussing sales estimates and strategies. Salesmen tried the various input materials to the model and found them

to be similar to the materials they already used in their current

methods. Initial reaction was one of caution, but the development

was continued by conducting an experiment where the new model was

run simultaneously with the current ones and the effects on sales

measured. The new model apparently improved sales and provided impetus for the airline to continue disseminating it to other sales

personnel, albeit cautiously. The authors conclude that the computer

model was really only a complement to the salesman's and manager’s

own information, not a replacement for it.

Although the system in this article is not complete, there are

a number of benefits to sources like this. First, even though the

report is brief and leaves out much key material on development, it

provides an overview of strategy. One might deduce from the article

that the strategy included definition of the problem, development

of a computer model, dissemination through structured face-to-face

seminars, high level management support, and a small, low risk field

experiment approach to testing the model with ultimate users. However, there are a number of shortcomings to the article also. It

is not clear from sources like this how the problem was defined early

on, how the authors and others were involved in development or whether

the model went through revisions. In fact, the lack of problems in

the development is conspicuous by its absence in the article. There
77

is an appearance that the new model was favorably received by users,

although management and user caution are emphasized a number of times.

One does not learn how well the model continued to be received by

other sales personnel and if it did in the end help solve the problem.

3 . 3. 1.2 A Detailed Case Study

A more detailed case study is provided by Lucas and Plimpton

(1972) on the development of an information system for the United

Farm Workers (UFW). Lucas used this case to test the creative system design approach he had developed over the years based on an organ!

zational change model from Kolb and Frohman (1970).

The report starts with a review of the creative approach and

then begins a narrative on what happened in the application. The

UFW problem involved record keeping and since the organization did

not have the expertise to build a system in-house, they turned to

outside consultants. With a lack of resources to support a study,

the UFW decided to use a student team headed by Lucas to design the

system. Besides experience for his students, Lucas saw the case as

an opportunity to conduct research on his design process.

Given the circumstances of the site, the team recognized early

that the system would have significant impacts for the staff. They

decided to survey a sampling of staff at most levels of the organization and conduct interviews with top managers. The team found that
interest in the system was strongest at the top management level and

that changing procedures with the actual users, especially older ones,

would constitute a significant intervention. Users had not had contact with computers before and the idea of bringing them in left them

with concern about their own replacement.

At this point, the team desired to continue in the diagnosis

phase of their approach and further study what users needed. However, additional pressure from UFW managers forced the team to also

begin the design phase as well. A deadline to complete the system

had been set by the managers in order to meet some records requirements for an outside agency. The team began making suggestions on

how to improve the records process at the same time they were learning

about it. Working between the managers and the users was further

complicated by the loss of the first liaison with the UFW and an ineffective relationship with the replacement.

The first meeting between the team and top managers was held

about this time. The team received its first official feedback since

the start of the project. The managers reported that the surveys

and interviews being done were causing disruption in the organization.

The change in students on the team was further confusing the staff.

The team recognized that the information gathering raised the level

of concern about the system, but the contact was considered beneficial

overall in establishing a trust with users and gaining valuable insight to system design. The team requested a new liaison staff member
from the UFW to improve the relationship between them. They needed

a "system champion” in the UFW.

UFW managers decide to assign a new liaison to the project and

this person had some amount of experience with computers. Unfortunately, the first task assigned to the liaison was to reevaluate the

contract with the team. Work slowed down during this review, but

project approval was obtained without much further delay. Diagnosis

activities were continued.

Soon the team began to plan for the first general staff review

of the team findings and suggestions for systems. Since this was

to be a major briefing, the team wanted to see top managers first

in a smaller pre-briefing meeting. Apparently as this was being set

up, top managers learned about the impending suggestions to be made

by the team. These suggestions did not include switching over to

a large computer system that management had originally expected they

needed. This caused great concern with the managers and they decided

to cancel the pre-briefing meeting. Instead they had a computer manu

facturer provide a demonstration of a system they might buy. The

team was instructed to redesign their suggestions with the proposed

computer system in mind. They went along with this, but insisted

that the UFW withhold final judgement on the computer. This apparent

ly happened.
The general review meeting was then finally held. It was a key

meeting for the team’s approach since they wanted to maximize input

from users on the system design. Also they wanted the union to accept

the idea of gradually increasing ownership and expansion of the system. Thus they were proposing a small scale system which would later

be expanded as it gained acceptance in the organization. All staff

members' came to the meeting and top managers began with assurances

that no one would lose their job to the system. The session lasted

a day and a significant amount of input was received. Interaction

with the staff was favorable, a positive sign for the team approach

and findings.

A number of smaller meetings were held following the general

review in order to obtain more comments and refine the design. The

union began considering how to procure the equipment it needed, but

later decided to find as much free equipment as possible. The team's

contact with the UFW was decreased over this time as the union took

on more responsibility for implementation.

One of the final tasks the team had in completing their research

objective was to survey the staff a second time. The purpose of this

survey was to assess the impacts and success of the design approach

they used. This had to be delayed however when the primary liaison

with the union was again lost. The team had built a good relationship

with their primary contact and it was expected that the contact would

facilitate distribution of a final survey. Now the team had to reap-
proach union managers and review the original research objectives,

which in their approach essentially constituted reentry into the organ

zation

Eventually the survey was administered to the staff. The most

interesting finding was that users perceived a low level of involvement in the design effort. The team found this hard to believe after

the amount of contact they had used in their approach. A few more

final review meetings were held between the union and the team after

this.

Case study reports like this one provide a rich level of detail

on development which is often missing in other articles. The relationship between the development team and the organization are highlighted and the reader can begin to understand what happens on a dayto-day level in a complex development process. An overall perspective

is also achieved in that Lucas' development approach, creative system

design, is included for structure. Even though the authors lose some

focus on the approach by laboring through the details of what they

did, they do demonstrate that a simple design approach and its benefits can be altered significantly by the realities of the site. This

is well emphasized in a very interesting table included by the authors

which lists major events, dates, and stage of the approach.
3. 3. 1.3 An Instructive Failure

An interesting case of a failure in system development is provided by Ruth (1978). He reviews the development and cancellation

of an advanced logistics system for the Air Force. His report is

a little brief on the details of development, but he does emphasize

some important factors that contributed to the failure.

The Air Force wanted to develop a very large information system

which would coordinate all functions involved in the receiving,

storing, and issuing of supplies. The system would include inventory control, stock and financial management, maintenance planning,

and distribution and transportation analysis functions. The design

would be based on the transactions which occur in these areas.

The intent was to process transactions immediately and store them

in a very large data base. To accomplish this, the Air Force depended on obtaining hardware technology which would be significantly

better than their current systems. The Air Force planned to save

a significant amount of money in personnel costs by switching to

the system.

The major task in the project was to unite the numerous information systems already in operation which handled different parts

of the supply process. This was a massive undertaking. Not only

were there different, separate machines involved, but there were

also five different computer languages in use. Thousands of com-
puter programs had to be rewritten. In addition, a new communications network had to be developed to coordinate all the systems.

The Air Force planned to develop all the components of the new system and then convert quickly to it.

The project did not proceed according to plan, however, as

numerous unforseen problems arose. Since many programs had to be

rewritten for the new centralized system, systems expertise across

all units had to be combined and coordinated. The size of this

effort overwhelmed the project. Another problem was the hardware.

The system thought superior for the job by the Air Force was not

obtained through the procurement process. Instead, the decision

was to use a computer which had never been applied to a problem

of this complexity. Later it was found that the system was unable

to handle the large data base and processing load desired by the

planners

Eventually, the magnitude of these problems forced the designers to abandon the plans for a quick conversion and try a piecemeal

development approach. This strategy also met with great difficulties. Congress finally canceled the project, some 15 years and

$200 million after it had been started.

Besides the technical problems discussed above, Ruth also finds

several other design and implementation factors which he feels contr

buted to the failure. These are:
• System Design vs. Organizational Design.

The new system was to be based on processing transactions

as they occurred. While Ruth considers this idea farsighted, he believes that in the end this objective was over emphasized in the design. The business needs of users

and their methods of operating became of secondary importance in the development.

• Radical change.

Numerous changes were planned in the development: in machines, languages, procedures, software, and eventually

in people, since the Air Force expected to replace staff

with the system. These changes affected every level of

operation and were of a very fundamental nature. The

size and speed of these changes presented a massive shock

to the organizations involved. The scale of the human

engineering problems was not well anticipated.

• Transition to the new system.

Conversion to the new system was to be abrupt and rapid.

The system was to be fully operational immediately. Unfortunately, as with most large scale developments, progress

did not match expectations and delays occurred. Meanwhile

the size of the development and the conversion strategy

pre-empted any on-going work with the old system. When the

new system could not be operationalized at once, the Air

Force had to return to the old system, with no improved per

formance resulting from the project at all. There was no planning for delays or for the need for backup in case of

failure

• Management tenure

The management of the project changed often. The same development problem was shared across several managers. Ruth

feels that, besides the continuity problems, the frequent

changes made it difficult for anyone to be held accountable

for the project.

Ruth recommends several improvements for future developments

of this size. First, he believes that the needs of users should be

the driving force in a development. In other words, designers should

adapt the system to the user environment and not the other way around
85

Second, projects of this size should probably be broken into several

pieces which can be implemented sequentially. Subsystem modules should

be institutionalized gradually. Rapid, massive, and abrupt changeovers have too many complicating implications for the organization,

especially with the staff. Also, the great potential for delays or

outright failures would seem to make an early major commitment too

risky. At least a backup system should be planned or maintained so

that the organization can continue to function even with problems

in the development.

Finally, Ruth believes that a stable management team is needed

to adequately control a project. Turnovers in personnel are to be

expected. However, managers should be sought that can make a time

commitment commensurate with the expected life of the project, or

at least with a portion of it that can be implemented as a unit in

itself.

3.3.2 Important Factors in System Development

Much of the literature relevant to systems development is written with the objective of identifying specific factors which are important to or greatly affect the success of a development. These sources

usually demonstrate the relationship of variables like organizational

characteristics, user involvement, or top-level management support

to system usage, adoption, or implementation. Factors are often defined as independent variables and system implementation (or success,
86

usage, etc.) as the dependent variable. Relationships between variables are used to provide lessons for developers.

Sources organized in this manner cover a wide range of factors

and rely on a variety of methods to identify, study, and report on

them. Everyone has their own perspective on what is important to

successful system developments. Three categories which help demonstrate

the variety available are:

• What methodology is employed in or underlies the article.

• How many factors are discussed.

• What factors are discussed.

Each of these areas are briefly described below and some examples

given. Some comments on the factor approach in general then follow.

3 . 3 *2.1 Methodological Differences

Sources which discuss factors important in the development of

a system have a variety of research methods underlying their themes.

These methods are used to identify factors and measure their impact

on project success. Some authors rely on literature reviews, personal

experiences, or surveys to study factors while others use some form

of controlled experimental design.

One example of the ncn-experimental side of the literature is

the often cited article by Radnor, Rubenstein, and Tansik (1970).
87

This is one of the early works on OR/MS implementation and it is commonly referred to in recent MS/MIS literature. The authors discuss

key factors having an influence on the implementation of OR/MS models.

They first present 17 factors related to successful implementation

which they extracted from a review of OR/MS literature. Some of the

factors listed are changes in management structure, the reporting

level of the OR/MS group, urgency of results, or need by the OR/MS

group to define results. The factors are not further defined.

The authors then present a model of OR/MS implementation which

is based on the 17 factors. Some of the relationships between factors are highlighted in the model and used to formulate propositions

about system development. For example, one proposition is that "the

level of implementation of a project depends on the client’s willingness to support the project's implementation, his ability to perform

the necessary new tasks, and the availability of money and personnel

to implement the project" (Radnor et al . , 1970, p. 976). Thirteen

propositions are listed using different combinations of the factors.

In studying these results, the authors were able to synthesize

three new factors which seemed to characterize key aspects in implementation. These were used to study implementations in 60 business

firms and 40 government agencies. The results of the study are presented in the article in the final sections. Each of the three factors, the type of environment surrounding the OR/MS implementation,

the nature of the client-researcher relationship, and the level and
type of top management support, is cross-tabulated with the degree

of implementation problems found in all cases. Chi-square statistics

are also computed and some are reported to be significant. Unfortunately, neither the factors or the problems are defined in the article

so that the reader can understand what was measured. For example,

client relations are listed as bad, medium, or good and implementation

problems are listed as high or low. It is difficult to place much

confidence in the results due to these problems.

An example of a more controlled design is described in an article by Lucas (1978a). He reports on an experiment to examine the

relationship between a number of different factors and the success

of a system implementation. Two groups in the same firm, one receiving a new system and the other keeping the old one, were surveyed

before and after the changeover was made. The dependent variable

Lucas used was salesman satisfaction with the system. This was used

as a surrograte for system implementation success since the firm's

managers had mandated the use of one system or the other. The independent variables were quality of information, benefits of making

a change, and costs of making a change. After the data were collected, Lucas ran a cross-lagged correlational analysis on the variables.

He found that user satisfaction with the new system had decreased

while user perceptions of benefits had decreased and of costs had

increased. These changes were significantly different statistically

from those of the control group. Lucas concludes that the lack of
89

success with the new system was caused by the changes in costs and

benefits from the old system.

While the results probably do indicate some facets of the change,

the situation was undoubtedly more complex than suggested in the article. Lucas himself admits that there were technical problems in the

new system. The lack of success in implementation' could thus also

be plausibly linked to the fact that the system didn't always work.

One wonders what other conditions were like, such as top-level management support, and whether they had an impact on success. Finally,

even though Lucas specifies the timing of the research relative to

the system change, it is hard to judge if the results reflect just

the transitory conditions of start-up or the stable conditions of

the longer term. A judgement of unsuccessful implementation and its

causes may be a little premature.

These two studies demonstrate a key point about the factor literature. The credibility and utility of the sources vary greatly, but

neither seems to be a function of the methods used to identify factors

and relate them to project success. A report on personal experiences

can appear just as credible or unbelievable as does one based on a

more rigorous experiment. This is because of problems with definitions and the unrealistic, or simplistic, structure of factor research

models. Terms like user involvement or top-level management support:

are rarely operationalized. While they probably do represent key

factors in successful implementations, it is difficult to learn what
behavior or actions are implied. In addition even if the factors

are defined or the observations on them described, the linkages between them and eventual success are long and complex. A factor could

be important to success one year and not the next. Attributing causality, such as demonstrated in the Lucas article, is very difficult

and often lends a feeling of structure and influence to a project

that isn't really there.

On the other hand, within limits, the structure and simplicity

provided by the factor approach is its major benefit. A complex process like a system development is reduced to a few key factors more

easily studied in one case or several. The value of studying the

factor literature comes in understanding that, at least in some cases,

factors like top-level management support are important. How, when,

and where they are important will vary and probably should not be

overly generalized by researchers or developers.

3.3 -2.2 Different Numbers of Factors

There is great variability in the number of factors which are

presented in literature sources. For example, in the practitioner

literature one can find numerous articles like one by Fronk (1978)

which lists five major problems in the development of data systems.

These include unending feasibility studies, lack of problem understanding by the developers, and continual design changes by the clients as they learn more about what can be done. The key factor he
emphasizes like most others is to involve the client early and routinely throughout the project. This might partly be accomplished

by more explicit contracting between developers and clients. He also

points out the difficulties that turnovers in staff can create.

More diverse and complex work can be found elsewhere, especially in academic presentations. Ein-Dor and Segev (1978) surveyed 10

years of scientific, managerial, professional, and trade literature

and identified 120 different factors related to the success of management information systems. In the 1978 article, they focus on a subset

of these consisting of 10 factors related to organizational context.

These include organizational structure (number of profit centers;

number of divisions), psychological climate (attitudes towards informa

tion systems; expectations from information systems), rank of the

responsible executive for the system, and the existence of a development steering committee. Through a series of propositions and support

ing literature citations, the authors relate these factors to MIS

success, which they define as use. One proposition is that "the more

mature the organization, the greater the likelihood of successfully

implementing MIS" (Ein-Dor and Segev, 1978, p. 1071). The implication

for developers is that mature organizations are more likely to have

rational, formal structures with explicit decision and control processes. Developers should be able to understand these organizations

more easily and quickly identify the processes needing an information

system.
92

A similar example is provided by Zand and Sorensen (1975) who

conducted a large study of the factors influencing success in applications of management science techniques and systems. Using a model

of intervention proposed by Lewin (19^7) which involves unfreezing,

moving, and refreezing stages, the authors developed a list of factors applicable to the different phases of development projects.

A survey was then sent to 391 management scientists who were asked

to identify and describe one successful and one unsuccessful case.

The resulting 140 cases were studied by the authors to associate their

list of factors with project success. A final list of 44 factors

was then produced, with most factors relating to technical problems

and relationships between top managers and system developers. The

importance of these factors in both successful and unsuccessful cases

led the authors to conclude that problem identification and building

good working relationships in early project stages were two of the

more important factors in the development of management science processes for organizations.

3. 3. 2. 3 Different Types of Factors

There is also a great variability across the literature in which

factors are most important to a successful development. In an often

cited article, Ackoff (I960) reports on four main problems in implementation that he experienced in 48 OR/MS projects, some of which

were system design studies. These cover a broad range (Ackoff, I960,

p. 262):
93

• Turnover: Reorganization of the company, changing

the persons responsible for the operations under study.

• Top-level Support: Lack of involvement in high enough levels

of management in order to enforce interdepartmental coordination.

• Sabotage: Attempts by individuals to capture the

project so that it can be used

to serve personal rather than organizational objectives.

• Resource Support: Economic pressures that lead to a general

reduction of expenditures for outside

services.

To alleviate problems like these he suggests to system developers and researchers that contracts and contacts be made and maintained with the appropriate management level. These contracts should

be breakable if development appears to be leading to an unsuccessful

implementation

Robey and Zeller (1978) concentrate on the relationship of user

attitudes, organizational structure, and implementation process to

successful MIS implementation. They report on the introduction of

an information system to two neighboring plants. One plant adopted the system and the other didn’t. The authors studied the user

attitudes, organization, and implementation process at each plant.

They found that users in the adopting plant had a more favorable attitude towards the system in terms of its ability to aid their personal

performance and in terms of the system's importance and urgency.

In addition, they found that the rejecting plant had a more complex

organization, in structure and authority, than the adopter. Interest-
ingly, the implementation process was apparently the same in both

plants. The system was designed and disseminated to users with no

formal introduction or involvement. In the successful implementation,

a key user with previous experience in the new system took the initiative and supported it. No one with enough influence did this in the

unsuccessful case. Developers might conclude from the article that

important factors in developments are the need to have users perceive

a personal gain from the new system, the need to tailor a system to

the organizational structure, and the significant positive effect

of a system champion.

Lucas (1976) studied the implementation of computer based models in 25 organizations in order to develop factors applicable to

MIS implementations. He examined a wide range of factors: model charac

teristics, attitudes towards the models, general attitudes, situational and personal factors, and decision style. Through questionnaires

and interviews, he attempted to relate these to successful implementation, although his study was conducted after the models had been

in use for over two years. He found that favorable user attitudes,

user involvement, management support, nontechnical decision styles

in managers, and model simplicity were strongly related to successful

implementation. The major implications he draws for developers is

that good attitudes towards a system are useful and should be cultivated at all levels of an organization. People resisting an implementa

tion should be made a part of the design team.
3. 3 .2.

Comments

The factor research literature applicable to system development

maintains an important, albeit sometimes weak, position relative to

other literature. To its credit, it provides a format for practitioners and researchers to disseminate key lessons or variables without

having to necessarily detail entire cases or prove more general underlying theory. Practitioners and some researchers can offer their wisdom

on important factors leading to success based on their own valuable

experiences. Other researchers attempt to define the key variables

and isolate relationships through more controlled research designs.

Even though the results of both approaches can be unrealistic or simplistic, the emphasis on few factors enhances communication with the

reader

However, there is emerging concern as to how useful factor style

literature is to both researchers and practitioners, especially considering the large amount of it that is available. Some ccraplain

that few commonalities can be found across the various perspectives.

Keen and Morton (1978, p. 196) list the five factors which they feel

are the only ones across the many studies and settings that have a

plausible, demonstrated linkage to implementation success;

• Top management support.

• A clear felt need by the client.
96

• An immediate, visible problem to work on.

• Early commitment by the user and conscious staff involvement .

• A well-institutionalized OR/MS or MIS group.

Some of the criticism is well deserved. The factor approach

does suffer from severe definitional problems. What is top management support in one case, is not in another. This, combined with

the large number of factors potentially relevant in any one case,

almost make each case unique and integration across them difficult.

In addition, the methods used in factor research also differ considerably since both practitioners and researchers contribute to the literature. Practitioners often provide loose supporting evidence for their

lessons, while researchers often add complex designs or statistics

to support their claims.

The basic problem in factor research seems to be that the complexity of system development does not lend itself easily to general

rules or patterns. Each situation can be unique and the relevant

factors important to success different (Duchesneau, Note 2; Keen

and Morton, 1978). Also, factors do not operate in isolation. Their

interaction with other numerous variables builds any one case into

a very dynamic situation where changes in relationships are likely.

These conditions are probably some of the major reasons for weak results in the factor literature (Ginzberg, 1978a).
3.3*3 Models for System Development

A significant amount of systems literature is devoted to models

which can be used to structure and guide a system development. These

models generally outline the major stages and tasks users and developers might perform, covering everything from the initial problem identification to the operation of a finished system. Development models

provide an overall view of the process not easily obtainable from

the other literature sectors reviewed in this chapter. Operationalizing the models for a specific application can be difficult, however.

Models have evolved from many different disciplines and applications and it is difficult to describe the field succinctly. One could

choose a number of different attributes to group models for discussion. For example, models might be categorized by source (behavioral

science, computer science, etc.), decision level they are useful for

(operational control, management control, strategic planning), decision methods they can work with (structured or unstructured), user

location (government, private sector, etc.), or combinations of these

(Anthony, 1965; Keen and Morton, 1978; Simon, I960). Rather than

try to cover all of the possibilities, a subset of three was chosen

which illustrated several different perspectives, mostly by source.

First, a section is presented on system engineering models.

Many of the systems concepts and procedures evolved from the engineering work conducted in military weapon system development in the early
98

1950 ’s. As weapons became more complex, along with the agencies developing them, engineering projects required a new approach which

would offer better overall coordination while allowing the development

of new, complicated components. Systems engineering became one of

the first models for developing systems, and it greatly influenced

many other succeeding projects and models.

A second section reviews models used by the government to acquire systems. Two models, one from the General Accounting Office

(GAO) of Congress and one from the Office of Federal Procurement Policy

(OFPP) of the Office of Management and Budget, are presented and compared. The GAO model applies to the acquisition of computer software

systems and the OFFP model applies to the acquisition of large scale

hardware or software systems. The influence of the earlier system

engineering models can be seen in these models.

A third section is then presented on development models specifically aimed at information systems. Two main subsections are included:

one on models often labeled as conventional or traditional and one

on models considered as alternatives to the conventional models.

Conventional models have evolved primarily through computer system

developments and concentrate on technical design problems. Alternative models attempt to increase attention to the human and organizational impacts of a new system and promote cooperative partnerships

between user and developer. A third brief subsection is included
99

on a problem both of these groups are concerned with: that of defining decisionmaker needs.

A final section summarizes the models presented in the above

sections and provides other comments.

3 . 3. 3.1 System Engineering Models

One of the original sources of concepts and procedures for

systems development can be found in military/weapons system programs

in the mid-19^0's to early 1960's (Hill, 1970, p. 124). At that

time weapons were becoming increasingly complex, especially aircraft.

For example (Hill, 1970, p. 125; Peck and Scherer, 1962, pp. 37-44):

• Weapons development began to involve the design and development of many different subsystems, requiring the

incorporation of a large number of technological advances

all at once.

• Subsystems and their components were becoming more highly

interrelated in size, shape, and input/output connections.

• Subsystems were becoming increasingly complex because

of the increasing number of components in them. This was creating reliability problems with system wide implications.

• Engineers and scientists were having to specialize on smaller parts of a weapon increasing the number of people

involved and the coordination problems.

The Armed Services also found it increasingly difficult to

develop weapons when development and production were divided among

several different military offices and contractors (Livingston, 1959,
pp. 84-86). It was recognized that better and more sophisticated

coordination would be needed to produce a "wholly functional" unit

(Hill, 1970, p. 124). Similar changes in thinking were also occurring in industry, some because of its contracts and others because of

its own needs (Hall, 1962).

Systems engineering was one of the concepts resulting from

the early thinking. It was fundamentally concerned with deriving

a system design that would achieve stated objectives (Chestnut,

1967, p. 37). However, many different ways of describing systems

engineering have evolved over the years and no one definition is

universal. General descriptions of it usually include phases like

the following (Flagle, Huggins, and Roy, I960, pp. 21-23; Hall,

1962, pp. 7-11; Hill, 1970, pp. 125-126):

1. Systems Studies (Program Planning).

This is an initial phase which often may not be included

in descriptions of systems engineering. In this phase,

decisionmakers develop the broad context for ensuing specific projects. Problems needing engineering and development are identified and resource allocations made.

2. Exploratory Planning (Project Planning 1).

This phase consists of a number of different tasks which

are performed in any sequence. The basic objective is

to identify the system development which will solve the

problem defined in the previous phase. Other work needed

to support a system development may also be identified.

Specific tasks in the phase include problem definition,

selecting objectives, systems synthesis, systems analysis,

selecting alternatives, and communicating the results.
3. Development Planning (Project Planning 2).

This phase is a recycling of the previous one except that

attention is now shifted to a specific system development.

Plans are formulated for the system, and the management

of the project. Design requirements and subsystem descriptions are specified so that the project can be adequately

defined.

4. Studies During Development (Action Phase 1).

A significantly larger effort is then started on development. Requirements are further detailed, studies are made,

and prototype models are produced and tested in the working

environment. Final evaluations of the system are made.

If the evaluations are favorable, then the system is transferred to users.

5. Current Engineering (Action Phase 2).

The primary development has now ended and an operational

system has been set up with users. Follow-up work is

done in this phase to further refine the system in light

of operating conditions. Development activities may be

initiated as needed to support changes in the system.

Other descriptions vary slightly from these phases. For example, Chestnut (1967) and Hill (1970) condense the five phases into

four: Conceptual, definition, acquisition, and operational. These

authors also point out that engineering design is often equated

with systems engineering. However, they believe that engineering

design is embedded within the systems process, covering activities

found in the definition and acquisition stages (Chestnut, 1967,

PP. 3, 36-37; Hill, 1970, pp. 124, 126).

Another key point made in most discussions about systems engineering is that descriptions of it specify functions performed

and not what a group does (Hall, 1962, p. 11). This point is made

in recognition that systems engineering involves numerous activities
102

and people. The organization and management of people and activities is a related but distinct process. Systems engineering management identifies the hardware, facilities, personnel, training, and

technical orders necessary in a systems project (Hill, 1970, p. 126).

This aspect will be discussed more in a later section of this chapter.

3. 3. 3. 2 System Acquisition Models in Government

The previous section in this chapter reviewed development models used in systems engineering and included some discussion on the

military weapon system procedures. These defense related models represent one major approach developed in government. Several others have

recently been designed, one from the U.S. General Accounting Office

(GAO) on computer models and one from the Office of Federal Procurement Policy (OFPP) on the acquisition of major systems. The OFPP

model is an update for procedures previously used in the military.

This section reviews and contrasts these newer models.

GAO Model

The GAO model was developed from a study conducted on the

acquisition of computer models (Comptroller General, 1975; Jue, Nowocin,

and Mandelbaum, Note 3)« Model developments were studied in organizations having access to government funds to develop or use models.

The agency identified 519 models in some stage of development and

closely examined 57 of them. While the developments represented
smaller projects and systems than of central concern in this dissertation, they did have objectives similar to larger scale systems.

They all were supposed to provide information for decision-making.

Also, many were to play a key role in spending large amounts of money

The GAO study thus provides some lessons for more complex system developments .

The GAO found management problems in over half of the developments. The problems fell into three categories (Jue et al.. Note 3,

p. 1 )

• Planning.

Planning problems included unclear definition of the problem, inability to obtain data, insufficient funds to com- plete the model, inadequate provisions for updating the

model for future use, inadequate planning of evaluation

procedures, and inadequate planning of documentation re- quirements .

• Commitment

Commitment problems included insufficient participation

of the intended model user in planning, and insufficient

knowledge of computer modeling techniques and applications.

• Coordination.

Coordination problems included inadequate monitoring of

model development, and inadequate user/developer

coordination

The GAO then surveyed all of the organizations to determine what

procedures would improve the acquisition of computer models. They

found few usable guidelines in the organizations they contacted and

decided to create their own five phase procedure. The phases are
104

(Comptroller General, 1975, pp. 15-23):

1. Problem Definition.

This phase concentrates on establishing characteristics

of the problem and the model needed to solve it. Users

should obtain management commitment to the model and define the changes that will be needed to support it. If

contractors will be needed, competition for them should

be implemented here and procedures for monitoring their

work established. Updating considerations should be made.

Overall, the phase should constitute a complete study of

the factors justifying development, what will be developed, how and when it will occur, and what happens when

development is over.

2. Preliminary Design.

This phase includes defining the model inputs and outputs,

logic, limitations, costs and benefits. Substantial revision of the problem in this phase could be grounds for

terminating the effort. Users should be confident that

all specifications are identified and will not need changing in subsequent phases. If needed, project should be

redesigned to accommodate the specifications developed

in this phase.

3. Detailed Design.

The developer now prepares the actual model, conducts

tests, and prepares user documentation. Periodic reviews

of the project should be made by users. They should also

decide if further development is warranted beyond this

phase.

4. Evaluation.

This phase is for more extensive testing of the model

under operational conditions. Criteria developed in earlier

stages are applied to model performance and the user determines if the model should be fully implemented.

5. Maintenance.

The user agency continues to monitor the model's performance and makes changes as needed. The developer should

remain available for assistance as agreed to in earlier

phases. If the model becomes outdated, support for it

should be terminated.
Two key points further stressed by the GAO are documentation

and separate budgeting of some phases. Documentation is needed in

each phase to demonstrate that the objectives have been met and to

prepare the basis for following stages. The GAO also considered it

important that the user have the flexibility to stop developments

which appear to be unneeded or infeasible. Thus they suggest that

users separate phases, especially problem definition and preliminary

design, from the others (Jue et al.. Note 3, p. 3). This could mean

as little as putting a decision point at the end of each phase, or,

more significantly, using separate budgets and competition for selected phases (Comptroller General, 1975, p. 15).

A number of considerations are missing in the GAO procedures.

First, there is little discussion of the overall management and organization of a development. A few of the key items, such as contracting,

are included as points to consider, but an overall perspective on

the preparation and organization of the people, resources, and processes is not provided. Secondly, the report treats development as

more of a structured, linear process than may be appropriate to the

situation. In complex, uncertain efforts, even relatively small ones,

defining the problem in the first stage and obtaining a managerial

commitment to a solution may not be possible, especially if the user

doesn't have the internal staff resources to do so. Contracting support may be needed early. Also, the process ignores the possibility

of cycling through stages, refining different parts of a model independently, or having overlapping stages. Thus it appears that users
106

must commit themselves to a significant amount of up-front work and

specification before they obtain outside help or test their ideas

with prototypes. Some reviewers of the GAO model also made this point

(Comptroller General, 1975, pp. 25-26). They felt the procedures

too structured for projects where flexibility and creativity are paramount.

Finally, the GAO model does not include much about what the users

do after the problem definition stage is completed. For example,

integration of a model with the user environment is rarely mentioned

in later stages, especially from the user viewpoint. Most of the

implementation issues discussed are found in the problem definition

stage, where it is difficult to see how they can be finalized so early.

More contact between user and developer would seem necessary.

OFPP Model

A more comprehensive approach to system development is provided

for in the policy directive recently issued by the Office of Federal

Procurement Policy (OFPP) of the Office of Management and Budget,

and discussed in a shorter pamphlet from OFPP (Office of Federal Procurement Policy, 1976). The new policy, A-109, is intended to apply

to all executive branch agencies. It is to improve the analysis of

agency mission needs, the identification of needs for a system, the

chances for innovative ideas from contractors, and the communication

process with Congress. Typical systems covered by the policy include
office buildings, energy demonstration programs, transportation systems, data processing systems, or defense systems.

The steps in the process are (Office of Federal Procurement Policy,

1976):

1. Mission Analysis.

Government agencies are required to structure their budget

data around missions and programs. The A-109 system acquisition process builds onto this structure by requiring con- tinuous mission level analyses. Deficiencies in capabilities or new opportunities within the missions may thus be

identified. A mission need statement is then created

to define the problem. This statement does not include

any proposals for specific means to meet the new need.

Once completed, the statement is then forwarded to the

agency head for approval. If it is approved, the statement

is also forwarded to Congress through normal budgeting

procedures. The intent is to have Congress review major

needs comparatively across agencies and debate the problem before commitments are made .

2. Exploration of Alternative Systems.

The agency establishes a system acquisition program and

explores alternatives to meet the mission need. A parallel program on non-system alternatives may also be created.

A program manager is selected and develops an acquisition

strategy and plan based on items from the approved mission

need statement. The acquisition plan is then used to com- municate plans to agency staff and other relevant outside

groups. The use of contractors is a major consideration

in the plan. A solicitation may be used in this phase

to acquire design concepts from outside consultants. The

solicitation should be constructed to allow creative,

innovative responses. A number of parallel short term

contracts can be let to develop competing ideas. Criteria

for selecting the most promising concepts that result are established.

3. Competitive Demonstrations.

When a demonstration of the evolving system concepts seem

feasible, the selected alternatives are submitted to the

agency head for approval. Evaluations of the other non- system alternatives should also be included. Competitive

demonstrations of approved concepts are then conducted.

The purpose of the demonstrations is to verify concepts,
108

check performance in an operational environment, and provide a basis for selecting concepts to enter full-scale

development later. Demonstrations may involve prototypes.

The program manager must assure that vendors can accom- plish full-scale development and production if they are selected at the end of this phase.

4. Full-Scale Development/Initial Production.

Proposals from contractors selected in the previous phase

are submitted to the agency head for approval. At this

point, competing design efforts may still be continued

if it is beneficial to do so. The program manager monitors contractor progress and coordinates changes from

original plans. The initial units produced are tested and

evaluated, normally independently from the agency user group.

Contractors should develop proposals for full-scale production.

5. Production, Deployment, and Operation.

Test results and proposals from the previous phase are submitted to agency head for authorization of full production.

As systems become available, they are deployed and made

operational. The new capability provided by the systems

then become a factor in the continuing mission analyses

described in the first phase.

OFPP emphasizes flexibility throughout the process. Iterations

through the phases may occur as more knowledge is gained on the problem, alternative solutions, or priorities, etc. The final production phase can be skipped if the system is not going to be produced

in quantity. A building would be an example of this.

Comparing Models

Comparing the OFPP model with the one from GAO demonstrates a

number of differences. Some of them are:
• The GAO model emphasizes more up-front specification of

the problem and the solution. The OFPP model provides for

more gradual specification. Early planning is emphasized in

the OFPP model, but it is recognized that all phases can contribute to an understanding of the problem and the system

needed. It encourages revision based on the increased knowledge of each phase. Final selection of the system comes

late in the process.

• The OFPP model is written from the user viewpoint throughout

all phases. Management and organization of the project by

the user is discussed.

• The OFPP model suggests more liberal use of contractors

to support the development. Contractors might be used in

early planning as well as later design stages. Several con- tractors might be used within the same stage to develop com- peting ideas.

• Commitment of the user to the system appears to be more gradual in the OFPP model. The GAO model emphasizes early

comprehensive planning and some level of commitment to an ap

proach. Later phases do not include competition between idea

If major changes occur in the problem definition or system

design, then the project should be terminated. On the other

hand, the OFPP model provides for change. It is recognized

that as designs and tests become more specific, it is likely

that changes in concepts, expectations and plans will be

needed. OFPP seeks to maximize learning about the problem

and the solution through competitive testing.

While the OFPP model is aimed at more complex systems than those

studied by the GAO , the two models are aimed at similar problems

and objectives. In each case, a need for a new set of procedures

is identified and either a model or a system is acquired. The differences between these projects are probably mostly in dollars expended,

people involved, and decisions and organization processes affected.

An increase in these factors results in increasing complexity of the

system. This may be the reason behind the flexibility of the OFPP

model versus the GAO model. This also indicates a difference between

smaller efforts and larger ones. Large scale system developments
need more flexibility. Early decision on designs are not easily

made or desired.

3. 3 . 3 *3 Information System Models

Models specifically designed to guide the development of informa

tion systems come from a number of different professional disciplines

For example, Keen and Morton (1978, p. 79) list six professional

groups from which have evolved six different viewpoints on information systems:

1. Computer Science.

Creates technology, both hardware and software. This

is a necessary but not sufficient contribution to information systems.

2. Management Science.

Represents the analytical viewpoint in structuring problems and develops the models so often necessary to drive

information systems.

3 . Behavioral Science.

Provides insights into the implementation process and

the human and organizational context of the system.

4. Data Processing Professional.

Builds the application systems the organization finally

uses.

5. Management.

Understands the realities of decisionmaking and thus which

systems can be effective.

6. Decision Support.

Focuses attention on building systems in relation to key

decisions and tasks, with the specific aim of improving

the effectiveness of the manager’s problem-solving process.
While discussion could be devoted to each of these groups and their

models, it seems that the major points can be summarized by a fewer

number of categories.

Several terms often used in the literature to group models

are conventional or traditional and alternative or evolutionary.

The conventional label is usually applied to those models in which

the developer is in control of the project and most attention is

focused on technical design problems. These models are commonly

associated with computer systems, especially clerical or accounting

types, since this was their origin. Data processing, computer science,

and operations researchers are the groups typically found using

the conventional approach.

Alternative models, on the other hand, refer to those development processes where users and developers share responsibilities

and more attention is placed on the human and organizational aspects

of design. They have particular application to higher level management decisions and situations where the information problem is more

unstructured. Behavioral and management scientists have been primary sources of these models (Keen and Morton, 1978, pp . 44-52).

The conventional and alternative categories are used in the

following sections to group development models, A further special

section is devoted to models used for determining decisionmaker

needs. The increased concern in the literature towards satisfying
112

user needs has prompted some authors to specialize in models

in this area.

Conventional Models

Many recent articles in the management science and management

information system literature identify models for system development

which are called traditional or conventional (Boland, 1978; Lucas,

1978b). Traditional models are aimed at situations where a system

is first designed by a technical team and then introduced and institU'

tionalized in the user organization. Developers and users have

fairly separate roles in the process. Developers are responsible

for assessing decisionmaking needs and designing the system, while

users just supply information and approve designs. Managers are

almost always seen in a passive role (Keen and Morton, 1978, p. 13).

Ackoff's early work in management information systems is often

cited as exemplifying the traditional approach. His process involves

five main steps (Ackoff, 1967, pp. B-153 -B-156):

1 . Analysis of the decision system.

Looking over the process of decision to be supported by

the system. Decisions are studied in relation to each

other. This step might clear up the information problem

without building a system.

2. Analysis of information requirements.

Define information needed by the decisionmaker and build

a model to produce it.
3. Aggregation of decision.

Group decisions with overlapping information requirements

under a single manager

4. Design of information processing.

Build the data collection, storage, retrieval, and processing subsystems.

5. Design of the control system.

Design flexibility and adaptability into the system. The

system will need improvements and changes.

The characteristics that make this model "traditional” appear to be:

• The emphasis on the designer versus the user.

• The lack of iteration through the steps.

• The apparent lack of contact with the user for feedback.

• The complete design and acceptance of a system previous

to any testing.

• The emphasis on technical issues versus organizational

or behavioral ones (Keen and Morton, 1978, p. 49).

• The ready acceptance by managers of the designs (Zand

and Sorensen, 1975, p. 533 )•

Patterns similar to Ackoff’s are also commonly found in MIS

literature which focuses on the use of computers. One frequently

cited source in this area is Davis (1974). He describes the life

cycle of computer systems and identifies three stages in their development. Briefly, these are (Davis, 1974, pp. 413-420):

1. Definition of the system or application stages.

Feasibility assessment, information needs analysis.
114

2. Physical design stages.

System design, computer program development, procedure

development

3 . Implementation stages.

Conversion, operation and maintenance, post audit.

Again, like Ackoff (1967), the emphasis of the model is on the designer. The user is in a subsidiary role until the time comes to

institutionalize the system. Also, the impact of change on the user

is not explicitly accounted for in each stage. Davis does recognize

that there will be impacts on users and that they can be significant

(Davis, 1974 , pp. 425-429). He suggests several actions the designer

can take to facilitate change for the user (Davis, 1974, p. 427):

• Encourage initiation of projects by the user rather than the

information systems department.

• Include user personnel in the project group where feasible.

• Establish a user advisory/supervisory channel such as one person or group to assist and guide the project.

• Hold informational and feedback sessions with employees

at each level who will come in contact with the system:

prior to design, at the end of design, prior to implementation, after implementation.

However, these seem to constitute a weak strategy relative to the

problem and only further emphasize the traditional nature of his

model

A third well known source in the traditional camp is Alexander

(1974). Alexander specifies three major steps in MIS development:
115

• Information gathering and analysis of the existing system.

• Design of the new system and preparation of a proposal.

• Preparation of system specifications and implementation

of a new system.

As with the other examples, implementation is emphasized late in

the process, the impacts of change on an organization are not explicitly considered, and the analyst is in charge. Alexander does

recognize that implementation is complex and that users should be

involved. However, he offers little guidance on how to get them

involved. For example, he advises that the system analyst should

structure the new system in order to introduce enriching qualities

to the user's job. This should make users more receptive to the

changeover (Alexander, 1974, p. 165). This does not seem to facilitate user involvement as it only reemphasizes the control of the

systems analyst.

Not all models labeled as "traditional" are like these three

examples however. In what seems a paradox in the literature, much

more elaborate development models which emphasize problem definition, user involvement, feedback and iterations are also placed

in the traditional camp (Boland, 1978, p. 888). An example of this

is the rather elaborate set of procedures for the development of

large, complex computerized information systems written by Bluraenthal

( 1969 ). Blumenthal recognized that information systems were evolving into more complex functions and that development problems were

not only technical, but managerial and organizational (Blumenthal,
116

1969, pp. 103-104). He discerned that even though much had been

written about planning large systems, little had been done to formalize a body of planning principles, especially in management science

literature (Blumenthal, 1969, pp. 7-8). He thus developed an elaborate process of his own which involves numerous committees, project

teams, design proposals, user reviews and acceptances, testing,

and evaluation. The heart of his model is a six stage process.

Within these stages, he delineate

1969, pp. 111-142):

Stage

1 . Preliminary

Analysis

2. Feasibility

Assessment

3. Management

Consideration

4. Systems

Implementation

and Control

5. Data-Processing • Organization

Activities •

6. Performance

Evaluation

14 specific procedures (Blumenthal,

Procedures

Pre-proposal

Proposal Preparation

User and Systems Staff Assessment

Additional Study

Presentation

Management Actions

Project Planning and Control

Completion of Functional Requirements

Preparation of Systems Specification

Programming and Testing

Conversion and Cutover

Hardware Planning, Technical

Assurance

Hardware Procurement and Installation

Systems Operations and Maintenance

Stages and procedures are linked to each other and often overlap

in their timing. Users are heavily involved in the early stages
as it is their responsibility to define the information problem and

develop proposals. Interactions through the procedures are expected

and some subsystems may be completed before others (Blumenthal,

1969 , p. 100). Technical designers have no more than equal importance to other members of the development team even in the mors specific design procedures. Blumenthal (1969, pp. 196-200) also acknowledges that systems must be adaptable to changes in functions, technology, or organization that may occur after development is over.

Blumenthal 's model and others like it would seem to be in a

different category than those presented by Davis, Alexander, or

Ackoff. The latter models reflect some of the earliest MIS thinking

and accomplishments which occurred in the system developments for

clerical, bookkeeping, or accounting functions of organizations.

These were among the first business activities to be computerized

(Blumenthal, 1969, p. 1; Karger and Murdick, 1977, p. 72). The

decision problems were structured, narrow, stable, and easy to convert to machine execution. While some claim that decisionmaking

was hardly a part of these applications (Keen and Morton, 1978,

pp. 2-3), it can at least be said that developing these systems

required less work on problem definition and more on design and

implementation. Keen and Morton (1978, p. 50) also suggest that

the novelty of the technology in these first applications often

overshadowed any human engineering considerations during development. Thus some of the traditional models can be said to reflect
118

the needs of these early applications and probably still would be

applicable to similar circumstances found today.

Blumenthal’s model on the other hand is not a simple, step-bystep process which separates user and developer, or handles only routine, structured decision problems. Much more emphasis is placed

on the problem formulation stages. Implementation is really on-going

throughout the development process (Chervany, 1978, p. 1?6). These

characteristics make the models more similar to the newer ideas for

MIS developments coming from behavioral scientists in the academic

literature and government decisionmakers in public sector. Behavioral

science models will be presented in the following sections.

Alternative Models

Another major category of models for system development has

been evolving from both behavioral and management sciences. The

models have several different names, but most often are called alternative (to conventional) or evolutionary models (Davis, 1974, pp. 405-

409; Lucas, 1978b; Urban, 1974, p. 224). Alternative models are

different from conventional models because they place more emphasis

on the human and organizational aspects of development. Problems

and methods from areas like social change, organization design,

systems analysis, organization behavior, cognitive processes of

decisionmakers, and resistance to change are used to structure de-
119

velopment strategies and reduce the over attention to technical design

problems given by other models (Keen and Morton, 1978, pp. 44-52).

Lessons from these areas are having an increasing influence on development models because it is being recognized that the success of a project

depends considerably on how well the process of change to a new system

is planned and managed (Youker, 1978, p. 12). The costs of ignoring

the human and organizational problems with change are being discovered:

more costly developments or reduced organizational performance overall

(Mumford et al., 1978).

Several examples of evolutionary models have been developed

from the Kolb and Frohman (1970) model of intervention into an organization. This model exemplifies many of the basic strategies behind

the recent models developed in the alternative category. Kolb and

Frohman expanded on models previously developed by Lewin (1952)

and Schein (1961) which specify three stages for organizational change:

unfreezing, moving, and refreezing. The Kolb/Frohman model consists

of seven stages and demonstrates the increased front-end emphasis

of the evolutionary approach (Alter and Ginzberg, 1978, p. 24):

1. Scouting.

The user and designer assess each other's needs and abilities to see if there is a match, and an appropriate organizational starting point for the project is selected.

2. Entry.

The user and designer develop an initial statement of

the project's objectives, and commitment to the project

is developed. They develop a trusting relationship and

a "contract" for conducting the project.
120

3. Diagnosis.

The user and designer gather data to refine and sharpen

the definition of the problem and its solution. They

assess available resources (including commitment) to determine whether continued effort is feasible.

4. Planning.

The user and designer define specific operational objectives

and examine ways to meet these objectives. The impacts of

the proposed solutions on all parts of the organization

are examined, and an action plan is developed which takes

these into account.

5. Action.

The user and designer implement the "best" alternative,

providing the training necessary for effective use of

the system in all affected parts of the organization.

6. Evaluation.

The user and designer assess the degree to which the goals

(specified during the Diagnosis and Planning Stages) were

met, and decide whether to work further on the system

(evolve) or to cease active work (terminate).

7. Termination.

The user and designer assure that "ownership" of and effective control over the new system rest with those who must

use and maintain it and that necessary new patterns of

behavior have become a stable part of the user's routine.

As seen in these stages, users and designers work as partners

and spend initial time developing their relationship. Much more

time is spent considering and refining the problem which a system

is to solve. The need to focus on the user-developer relationship

has been echoed in many other sources as well. Some examples are

Churchman and Schainblatt (1965) with their "dialectic of implementation", Mitroff and Emshoff (Note 4), Mitroff, Williams, and Rathswahl

(1972), and Mason and Mitroff (1973) with their strategies for ill-
121

structured problem solving, and King and Cleland (1975) with their

decision needs analysis.

Lucas has perhaps made the most extensive use of the Kolb/Frohman

model in his work on system development. His earliest model, called

creative system design, involves a combination of the Kolb/Frohman

stages and his own user-oriented concepts. He emphasizes the importance of user control, user interfaces with the system, user criteria

for evaluating the system, and the gradual commitment over time

of a user to a design (Lucas, 1974b, 1978b). Lucas provides an example of these concepts in a development for the United Farm Workers

(Lucas and Plimpton, 1972) and this is reviewed as a case study in

Section 3 *3. 1.2.

His latest model, called evolutionary design, builds upon the

creative design model and revises the series of stages (Lucas, 1978b).

The major change in the evolutionary model has been the increased

emphasis on interaction and feedback between user and developer,

especially in early stages. The stages are as follows (Lucas, 1978b,

pp. 44-46 )

Stage Activities

1

Inception • User suggestions for a new system

2. Initial Groping • Response from developer

suggestions

on initial

• Output (simple models,

simulated system)

dummy report
122

3. Mutual Progress • Collaborative Development

- define problems

- design system

- cycle through these

- test system

4. Conversion • Transfer of ownership

• Conversion

• Installation

5. Maturity • Operation

• Changes in design

As seen in the model, users produce sijggestions and developers

react quickly with models, dummy reports, or simulations based on

the suggestions. This procedure gives users something concrete to

react to, educates the designer more quickly, and helps clarify the

problem to the user. Although this cycle isn’t repeated in the model,

Lucas expects it to be the basic pattern of interaction between user

and developer throughout the project. Lucas also sees this model

as mostly applicable to cases where the decision problems and system

objectives are initially unclear.

Information Needs Analysis

Most of the literature devoted to information systems specifies

the need to satisfy the user requirements for information. While

this seems an obvious point, too many developers have experienced

the failure of MIS projects specifically because of too little attention to this issue. Much attention is now devoted to the front-end

stages of development where the initial work must be done to discover

these requirements. Information needs or requirements analysis is
123

now a specialty topic witin the various fields devoted to information

systems development. For example, Taggart and Tharp (Note 1, 1977)

identify and review 50 different models for information needs analysis. They also provide guidance on how to best select one for a specific situation.

There appears to be two different schools on needs analysis, one

which uses the data analysis method and one which uses the decision

analysis method (Boland, 1978; Davis, 1974, pp. 409-411; Munro, 1978).

The data analysis method has the following steps (Munro, 1978, p. 34):

1. Examine all reports, files, and other information sources drawn upon by the manager.

2. Discuss with the manager the use of each piece

of information examined.

3. Eliminate unnecessary information.

4. Determine unsatisfied information needs through

interaction with the manager.

The data analysis method concentrates on the existing structure of

decisions and gaps. It is implicitly assumed that the decisionmaker

knows what he wants (Munro, 1978, p. 38). This method would thus be

more applicable to situations where decisions are defined, stable, and

routine

In contrast, the decision analysis method has more of a normative

perspective and is more relevant to situations where there is uncertainty in the decision problem (Munro, 1978, p. 38). It includes

procedures to examine what decisions are made or should be made, and
124

how they should be made. Munro (1978, p. 36 ) lists the following

steps for the decision analysis method:

1. Determine major decision responsibilities through

discussion with the manager.

2. Determine policy and organizational objectives

relevant to decision areas identified.

3 . Determine specific steps required to complete

each major decision.

4. Develop a model (flowchart) of each decision.

5. Examine flowchart to determine information re- quired at each step in the decision.

King and Cleland (1975) elaborate on these procedures even more.

Generally, the decision analysis approach does not ass'ome things

are "as they should be" and emphasizes broader analyses. Normative

and descriptive information flow models are produced and gaps identified. Decision procedures as well as the decisions themselves may be

restructured after the analysis. A test conducted by Boland (1978)

showed this procedure to be better at identifying problems and stimulating ideas for appropriate system designs than the data analysis

approach.

3 . 3 . 3 . 4 Comments

The last several sections have reviewed some of the different

types of models available in the literature for systems development.

Three main categories have been used to group models: systems engineering models, system acquisition models in government, and informa—
tion system models. System engineering models evolved from the

need to improve the development of military weapons systems. Many

of the well known concepts in systems development were established

in these efforts and models in other fields have been influenced

by them. Government system acquisition models were established

to help guide agencies through the development of large scale systems. The models reflect the agency environment: the need to rely

on outside contractor support in a development, the need to satisfy

Congressional and budgetary scrutiny, and the need to promote competition between alternatives before reaching final production stages.

A final section reviewed models specifically aimed at developing

information systems. Subsections were presented on conventional

models, which evolved primarily from computer system applications,

and alternative models, which were designed to increase attention

on the human and organizational impacts of a system. A final section was included on information needs analysis models, which are

specialty processes aimed at defining decisionmaker needs.

There are a number of similarities between the various models:

• All models have some form of initial planning stages where

problems are studied, plans are developed, initial ideas

on system design are suggested, and approvals from users

are obtained. Most sources in the information system

fields recognize the importance of information needs

analyses at these early stages.

• All models have stages for testing and evaluating a system

prior to final full scale operation. The concept of using

prototype designs in these tests has long been accepted in

engineering and government models and it is now appearing more

in information systems literature as well.
126

• All models emphasize to some degree the need for decision

points where users provide approval for continuing work

or acceptance of a design. Some models locate these points

in all stages, while others only occasionally mention them.

• All models of course emphasize the importance of user participation, although, as with decision points above,

this interaction varies considerably across the different

processes

There are also a number of differences between models and the

following list covers some of the major ones:

• Design flexibility/user commitment to designs.

There are some differences between models on when technical specifications are accepted and finalized. The OFPP

model is perhaps the most flexible in this respect since

technical changes are routinely expected, competition between

ideas is encouraged, and subsystems are developed as independently from one another as possible. The OFPP models attempts

to keep the design options open as long as possible into a development. On the other hand, the GAO model suggests commitment to designs very early in development. Projects should

be abruptly cancelled in later stages if the initial design

specifications aren't met. Conventional information system

models are somewhere in between: user commitment usually

comes very late in development after the technical team

completes its work, and the technical team maintains less and

less flexibility in design over time. Alternative models

emphasize gradual user commitment to a design.

• Development process - project management.

System engineering models recognize that two separate

processes are active in a system development: the actual

strategy and tasks, and the management of these. These two areas are recognized to some extent in the government models,

primarily because contractors are usually involved in developments and they must be managed by the user. Information

systems models hardly recognize these issues at all.

• User activities versus developer activities.

Each of the models is written from a slightly different

perspective. Government acquisition and system engineering models generally have the user viewpoint in each stage.

The user is in control of the project. The conventional
information models take the opposite viewpoint and emphasize

developer control of the project. Users primarily just supply

information and allow the developer to have control of the

process. Alternative models have emphasized a more equal

partnership between user and designer, since both must be

educated about what the other is doing. These models may

overestimate the ability of users to participate equally

in the various development stages.

• Use of contractors.

Government system acquisition models include numerous

considerations related to the use of contractor support.

This reflects the fact that government agencies usually

don’t have the internal staff resources to develop

large scale systems on their own. Information system development models have little if any mention of this kind

of situation, except briefly in reference to computer system

vendors. It seems likely that, with large scale systems

most users, both public and private, will interact with a contractor at some point. The lack of attention to this in

information system models may indicate some lack of insight

as to how systems really get built.

• Human and organizational impacts.

Alternative models for information systems development

place the most stress on the impacts new systems will have

on organizations; conventional models concentrate more

on technical design problems. It is the lack of human

engineering concern in these models that prompted the

search for alternative models. Government acquisition models have their emphasis on organizational implications in early stages. System engineering models do

not specifically mention these kinds of issues directly,

but they are a long standing component of the systems

tradition

Each of these models has its own merit. Developers needing

guidance on strategy can take lessons from them and their applications regardless of the various differences. A model's utility

will be based on the match between its strategy and the prospective

application
128

The choice would probably be different for the decisionmaker

levels like operations, management control, or strategic policymaking (Anthony, 1965), and depending on whether the decision procedures

are structured or unstructured (Gorry and Morton, 1971; Keen and

Morton, 1978; Simon, I960). Other factors affecting choice would be

how big and complicated the project will be, who will run the project, and the degree to which the system will be a new design versus

a modest well known one (Thompson, 1974, p. 30).

3 . 3.4 System Designs

A final sector of the systems literatures is concentrated on

the design aspects of systems. Generally, this literature is divided into two categories: sources discussing design issues at a general level, and sources presenting specific designs for example

or real application. Many of the sources presenting general issues

tend to focus on one basic point: that one particular design in

not right for all situations. Developers must identify the critical

design factors, such as the user managerial levels, the complexity

of the information problem, or style of decisionmaking, in order

to select the appropriate design. These are lessons applicable

across most situations.

Specific designs, on the other hand, focus on one design and

its application. These examples are most useful for understanding

how the information problems were solved in a specific case. The
129

lessons offered in other systems literature sources can be observed

first hand with the actual result. These designs can be stimulating

in that the approach may offer insight to another situation exhibiting similar characteristics. Generally, however, the transferability of designs in this sector of literature is low, supporting the

idea usually discussed in other design literature that each situation is unique and requires a closely tailored design.

A note should be made about one source of confusion in the

design literature. This is the often side-by-side use of design

as a noun and as a verb. Design, when used as a verb, means the

actual activity of specifying and developing the system components.

Design as a noun clearly refers to the resulting specifications.

This can be confusing, especially in exploring the literature.

Sources discussing system design can mean either the process of specifying the design or the design specifications. In this section,

design will be used as a noun.

The following sections review in more depth the two categories of system design literature introduced above. An example of

a design has been selected for the section on specific designs.

A closing section conments briefly on the two sections.

3 . 3.^.1 General Issues in System Design

One sector of the literature on system designs concentrates

on general issues or lessons developers should consider in the pro-
130

cess of creating a specific system. The most common point made

in these sources is that one particular system design is not right

for all situations. There are many factors in a user organization

which will influence the kind of design needed, and these factors

vary considerably. Specific situations are more likely to be unique

in design requirements, and developers must acknowledge this by

closely tailoring the design to the user.

Perhaps one of the more widespread frameworks used by authors

to help categorize these differences is the framework developed

by Gorry and Morton (1971). In this framework, Gorry and Morton

list two dimensions which can be used to characterize the kind of

information situation which developers may be encountering. One

dimension defines the purpose of the system application and is divided into three categories adapted from work by Anthony (1965): strategic planning, management control, or operational control. Gorry

and Morton suggest that the information requirements for each of

these uses are quite different, and thus the system design must

be different. The second dimension loosely defines the method of

decisionmaking or information usage and is divided into three categories adapted from Simon (I960): structured, semi-structured,

and unstructured. Structured methods of decisionmaking imply that

information needs are well known, routine, and predictable. Systems

to meet the information needs of a structured situation are most

likely straightforward and quickly defined. Unstructured methods,

on the other hand, imply that information needs are unknown, chang-
ing, or unpredictable. Situations which are unstructured tend to

be found in upper level management where decisionmaking is complex

and involves many factors. Systems serving users at these levels

have to be much more flexible than in structured situations. The

main requirement may be that they are quickly changeable by the

decisionmaker in order to fit the decision problem at hand. Recent

thinking on systems like this can be found in the literature devoted

to decision support systems (e.g., Keen and Morton, 1978).

Gorry and Morton combine the two dimensions to form a matrix

which can classify the information need and thus assist developers

in determining what kind of information system design is appropriate. Many authors have also adopted this approach. Usually, structured situations are associated with the operational control level.

Here decisions are routine and repetitive. Sophisticated, dynamic

system designs are rarely needed. Similarly, unstructured situations are usually associated with higher decisionmaking levels where

needs can be fluid. Systems to meet these requirements are becoming

more of interest in the literature (Gorry and Morton, 1971, pp.

61-62) and can be seen in the aforementioned emerging decision system literature.

Other approaches to classification like that proposed by Gorry

and Morton have been taken, although they still emphasize the same

points. For example. Rich (Note 5) refers to the differing requirements dictated by the various decisionmaking styles that can be
encountered. Managers have different information utilization patterns and problem solving styles. One system can not easily match

the requirements of different people or different situations. Developers must be sensitive to the characteristics which contribute

to these differences and bring these considerations into the design

process (Rich, Note 5, p. 27). Rich also points out that not all

needs can be met by an information system, clearly recognizing that

there are limitations for designs which have to be recognized by

both developer and user (Rich, Note 5, p. 27).

A more humorous but equally realistic treatment of design issues is provided by Gall (1975). Gall lists numerous axioms about

systems with a heavy emphasis on why they often fail. Among his

many axioms can be found some that relate to design:

• A large scale system, produced by expanding the dimensions

of a smaller system, does not behave like the smaller

system (p. 22).

• A system is no better than its sensory organs (p. 40).

• A complex system that works is invariably found to have

evolved from a simple system that worked (p. 52).

Implications for design from these principles are useful and often

supported by others. For example, the second axiom listed above

implies the importance of input and output functions in the system

design. These may often be the weakest link in the design, a point

well recognized by others (Lucas, 1975b, p. 112; Malvey, 1977, pp.

191-211; Waller, Kemp, Scanlon, Tolson, and Wholey, 1976, pp. 13-
133

14). A rather interesting example of the third axiom, that concerning complex systems evolving from simple systems, is illustrated

by Harrison (1978) who describes project control systems. Harrison

suggests that large projects requiring control of multiple subprojects present complex management problems. He goes on to describe

the major concepts and components in project control systems and

reviews ones his firm has successfully applied. These systems

are themselves complex, usually involving computerized routines.

However, he describes one "special” technique they have found essential in the design of control systems: the use of a manual procedure

of reporting cost changes which consists of writing changes on pieces

of paper and delivering them directly to decisionmakers. This must

be the one area where system designers found it difficult to replace

a simple procedure with a complex computer program that produces

impressive mechanical reports.

Lessons such as these are applicable across many situations,

but application in a specific case requires the serious consideration of developers. Many of the suggestions about system design

are essentially describing a non-event; they don't specify an action

or behavior which is readily adaptable into a development process.

These ideas can be turned into a process, however (Thompson, Note 6);

this can provide useful suggestions on how to structure the development to avoid the problems implied by the authors.
3. 3. 4 *2 System Design Examples

A wide variety of system design examples can be found in the

literature. They range from general descriptions of certain categories of systems to more detailed discussions of specific systems

designed for specific applications. Each type of example can provide insight to what designs look like and associated important

development factors. However, experienced developers will find

only the most detailed designs useful for guidance in their own

respective situations.

One source of design examples is the literature devoted to

general technical design characteristics of selected types of systems. These are most commonly found in areas related to information

systems, especially involving computer machinery. One example is

provided by Matthews (1971) who discusses the design of computer

based management information systems. Matthews first reviews the

system life cycle stages often found in computer related literature

and then he describes general designs for systems, including descriptions of specific machinery components and how they work. Sources

like this are most useful as introductory material or textbooks

for general design courses. Experienced system developers will

not usually find much information from which to derive guidance

for specific applications.

Slightly more detailed design examples can be found that further illustrate applications of the kinds of systems introduced
by Matthews. Sources can be found which provide an overview of speci

fic systems that exemplify those in the field. For example, Palmer

(1973) reviews a number of library information systems used for

cataloguing, ordering, managing, and circulating library materials.

While focusing on descriptions of the designs, Palmer also includes

some details on development aspects and whether the designs met

the expectations of users. Even though the examples are brief and

the systems fairly small, readers can begin to assimilate important

factors for development by observing the problems Palmer describes

and how developers reacted to them.

A similar but briefer example in another field is provided

by Alter (1977) who reviews categories of decision support systems

currently being designed. Seven types are discussed: file drawer

systems, data analysis systems, analysis information systems, account

ing models, representation models, optimization models, and suggestion models. Alter describes the general designs of these systems,

including hardware and software components, and how the systems

operate. Some information is also presented on the problems of

designing and implementing these systems.

Other examples similar to the one by Alter can also be found

in other literature areas. Many of these occur in trade publications where system managers describe a specific system used in their

company. For examples, Kronenberg (1967) briefly describes the

evolution and design of an information system used by Weyerhauser.
Overall, sources like these provide a quick glance at specific appli

cations and in some cases offer limited insight to key development

problems encountered. Mostly, however, the examples are brief and

useful mainly as an overview of where applications are occurring.

It is difficult to derive and support key development factors from

these sources.

More detailed and useful examples can be found in sources such

as Keen and Morton (1978) who review the emerging field for decision

support systems (DSS). Decision support systems generally are inter

active computer systems designed to closely support a manager’s

problem solving style and need. In their book. Keen and Morton

( 1973 , pp. 99-166) review six systems in operation that exemplify

the DSS approach, including a portfolio management system, a financial planning system, and a management information system. Generally, descriptions of these systems concentrate on the technical design characteristics. They review data bases, system operation,

and input and output displays. Other components of the systems,

such as for maintenance, updating, or training, are not described

in similar detail.

Some of their examples are notable for additional insight on

development however. The most interesting example is presented

for the portfolio management system, where one successful development is compared with one less successful development (Keen and

Morton, 1978, pp. 101-126). In both cases, a system was designed
and implemented in a bank. However, one system met the expectations

of users and improved their operations while the other system was

less successful. From the brief details supplied in each case,

readers can identify several important factors influencing the development success for these systems. In the less successful system,

several factors stand out:

• The heaviest users, although limited, were the people

who participated in design activities. The lack of a train

ing program for those not involved contributed to their

lack of participation.

• A prototype system was a useful intermediate step in the

design process. The low risk of testing a system without

making a full comnitment increased the number of users participating in the tests. However, it was also found

that early participation did not guarantee later adoption

of the system. A rapid changeover was not possible.

• Conflicts between the separate groups of users and designers, mostly a function of their different technical backgrounds, led to decreas.^1 user participation in testing.

• Use of the system required changing existing organizational routines. Little effort at overall change reduced

the incentive to use the new system.

In the more successful case, several other factors can be identi

fied, some of which contrast the conditions in the first one:

• One critical staff member became a system champion and

pushed for the use of the system.

• Top level managers supported use of the system and created

incentives to make the changeover to it.

• Top level managers considered the intangible benefits

or future spinoff effects of use to be an important justification behind their support.

• Training was provided to integrate the sytem into day

to day activities.
138

Examples like this one by Keen and Morton provide insight to

both design and development characteristics which other developers

can understand and possible apply to their own situations. While

they are brief descriptions and leave out many details, they begin

at least to demonstrate development implications for certain designs

and to confirm in actual settings development principles identified

elsewhere

An example of a single more expanded design is provided by

Nay, Scanlon, Graham, and Waller (Note 7) in their report on a system

developed for the Law Enforcement Assistance Administration (LEAA).

LEAA wanted a system to support the monitoring and evaluation of

programs undertaken by its grantees. Viewing the information system

as a knowledge production process. Nay et al. describe the development, design, operation, and performance of the system that resulted. While the descriptions of the design are general, this case

is interesting because it;

• Demonstrates a "software” type system.

• Demonstrates a design for situations where there are changing data needs, many unknowns about field operations,

and myths about what will be learned using a new system.

• Provides descriptions of development activities along

with design descriptions.

• Provides examples of products and their impacts.

• Provides evaluations of the resulting design in terms

of cost, time, and performance.

• Shows how the design changed with experience.
Nay et al. also include brief discussions on whether it is

beneficial to create a system as a distinct unit in an organization

with its own staff and on how the project worked as essentially

an in-house development.

Examples like this are perhaps the most useful in the design

related literature. They give details beyond simply the design

requirements and technical details, including specifics on the associated development, testing, operation, and maintenance functions.

Design concepts and specifications can be tied to other factors

and traced over time, adding a dynamic perspective to the example.

Experienced developers will find these examples interesting and

illustrative and be able to use them as models for future applications. More of these examples are needed across all sectors of

the literature.

3.3.5 Conclusion

The previous four sections have reviewed the systems literature

in detail with a specific emphasis on what materials are available

for guiding system developments. The literature was divided into

four main areas; case studies of system developments, important

factors of system development, models for system development, and

system designs. Illustrations of sources in each of these areas

were included.
140

The following sections review important problems for system

development that are currently not well discussed in the literature.

Three main problems have been identified in the review of literature

areas: the lack of a comprehensive view of development, the need

for a more dynamic perspective of the evolving, changing nature

of development, and the need for a more comprehensive view of the

process by which users decide to support, accept, and eventually

implement a system. One section is provided for discussing each

of these problems. In addition, a final section reviews the key

problems as seen by authors in the systems literature. The structure used to describe the systems literature in this section will

also be used to discuss these development problems.

3.4 KEY PROBLEMS IN THE SYSTEM DEVELOPMENT LITERATURE

3 . 4.1 Introduction

This section of Chapter 3 discusses the key problems for system

development found in the literature. The presentation is divided

into two main areas. First, the problems for development which

are not adequately recognized in the literature are discussed.

Three main problems are identified: the lack of a comprehensive

view of development that integrates the various considerations potentially applicable in a systems project, the need for a more dynamic

perspective of the evolving, changing nature of development, and

the need for a more comprehensive view of the process by which users
decide to support, accept, and eventually implement a system.

Next, the problems which are identified as research needs in

the literature are discussed. Three major needs are identified:

the need for more research on the management of system development

projects, the need for frameworks which can help developers diagnose

the management problems of their projects and find the guidance

available in the literature, and the need for more empirical evidence which demonstrates how management techniques are used in actual situations.

Finally, sections are included which describe the approaches

used in this dissertation in response to these problems and needs.

Descriptions are included in all of the sections in order to more

closely identify the dissertation approach in relation to the specific problem discussed in each section.

3 . 4.2 The Multi-faceted Nature of Development

The developer wanting to utilize the literature as a source

of support for a new development will find it difficult to obtain

a comprehensive view of the many important facets that need consideration. This is because much of the literature is segmented and specialized around selected issues, models, or designs. For example,

typical discussions are those such as development strategy alternatives (Lucas, 1978b), system designer-use relationships (Boland,
142

1978; Patterson, 1977), or the five most important development problems from the user viewpoint (Fronk, 1978). Each of these perspectives provides valuable insight to a particular aspect of system

development, but few sources attempt to interrelate these considerations in one place. It is interesting to note along this line that

researchers themselves are now finding the need for more unifying

frameworks (Bostrom, 1978; Ein-Dor and Segev, 1978; Ginzberg, 1978a;

Zimmerman, 1977).

This section will review in more detail the kinds of perspectives taken in the literature and then discuss an approach which

further enhances the contribution of each.

3. 4. 2.1 Limited Perspectives in the Literature

The systems literature overall is both broad and detailed on

the many issues to be considered in a development. Readers can

find numerous discussions on any range of topics, and those devoted

to a comprehensive view will eventually be able to cover the important issues. However, the literature is segmented in certain ways

which lead to a high degree of specialization in the presentations.

Generally, the literature is divided by three dimensions.

The first is the main facet of development that authors select

to discuss. As shown in the preceding descriptions of the literature, there are four prevalent perspectives used when discussing
development: a case study approach which concentrates on the sequence of events, a review of key factors in a development, models

of development processes, or descriptions of system designs. Each

of these provides a distinctive viewpoint of development and has

its limitations for providing a comprehensive view:

• Case studies of specific developments offer a detailed

view of development, usually focusing on the sequence

of major events. However, they are often poorly organized

or without an overall perspective on strategy. Many are brief and leave out the complex series of events between

initiation and final design. Major lessons from these

experiences often appear to go beyond the evidence presented in the case.

• Factor studies identify and examine selected variables

found to be important in system development. Commonly

discussed factors include the need for user involvement

and top level management support. Some sources further

attempt to establish relationships between factors, usually focusing on the linkages to system adoption or usage.

Factor studies offer more controlled and detailed examinations of variables and causal linkages than found in

case studies. However, the complexity of development

usually leads to weak study results (Ginzberg, 1978a).

Claims made from the research give the appearance of structure where in reality little exists.

• Models of system development provide more overall structure to a development than usually found in other literature areas. Models are usually divided into several stages

covering the entire development process. Each stage can

be further specified by its respective changes in design

or strategy. The major problem with models is operationalizing them in a specific setting.

• System design literature provides descriptions of designs

used in specific situations or general design issues found

to be important in an experience. These sources offer

useful ideas on what a design looks like, but rarely include guidance on how the results were obtained.

The systems literature is also divided by the range of variables considered in the specific source. For example, the use of
144

contracting to support development is a common, well recognized

feature of development models used or promulgated by government

policymakers. Models proposed in the behavioral and management

science literatures rarely consider this as a critical component

of a development project. Other differences can be found:

• The emphasis on user involvement in development is not

often compensated by similar discussion of the design

teams themselves, which might include users as well as other outsiders.

• Development models or important development factors may

be presented from only one perspective, that of the user

or that of the developer.

• Government development models emphasize competition among

design concepts or subsystems and generally recognize

the uncertainty pervading a complex development. Other

models, particularly in the behavioral and management

sciences, seem to portray a simply implemented process

in which everything is done correctly and logically the

first time.

• The concept of using prototype models is only beginning

to emerge in behavioral science based systems literature

(e.g., Fronk, 1978; Lucas, 1978b; Moore and Byrd, 1977;

Zimmerman , 1977 )

• Behavioral science development literature often overemphasizes human engineering problems and minimizes the concern

on how to actually manage a development. Computer science

literature concentrates more on the technical aspects

of design than the human engineering components (Keen

and Morton, 1978, p. 3^)*

• Literature concentrating on factors important in development may only discuss a limited number of factors, such

as user attitudes, top level support, resources needed,

or the kind of user environment needed. The relationship

among factors like these is often left undiscussed.

Finally, the literature is divided into sectors according to

the kinds of systems of interest to the authors. Literature sectors
145 I

1

can be found for weapons systems , management information systems

(MIS), computer systems, hardware or software systems, decision

support systems (DSS), and many other kinds of systems. While the

specific applications are quite different, the techniques and lessons in each sector tend to be similar. Thus, developers needing

guidance for an information system development may benefit from

a wider review of different literatures where analogous problems

are discussed.

3. 4. 2.A More Comprehensive View

The complex, multi-factored nature of system development is

not well represented in the various literature sectors described

above. On the other hand, the complexity is difficult to present

since there are so many factors to consider. Sources concentrating

on selected factors are able to provide supporting detail not feasible in a more general discussion.

What is missing is the occasional, comprehensive view which

attempts to interrelate the various factors in one location. Without these types of sources, readers of the systems literature must

cover a wide range of material and assimilate for themselves the

diverse, critical aspects relevant to their own situation.

Within the information systems literature, the best approach

for tying together the various perspectives appears to be the combina-
146

tion of two or more types of sources: a case study with a model,

a development model with important factors by stage, or the examination of selected factors in a specific case. A number of examples

of these can be found in the literature:

• Lucas (1974b) describes his creative system design model

and includes several brief case studies as illustrations.

• Lucas and Plimpton (1972) describe the creative system

design model and illustrate it with a detailed case study

of a development for the United Farm Workers (see section

3. 3. 1.2).

• Zand and Sorensen (1975) describe a development model

based on Lewin's (1947) model of change and identify factors important in each stage (see section 3. 3 .2.3).

• Eveland, Rodgers, and Klepper (1977) describe a model of

adoption and then illustrate it with case study details

of system developments in eight locations.

• Alter and Ginzberg (1978) describe the Kolb and Frohman

(1970) model for intervention in an organization and cor- relate important development factors with each stage (see

section 3. 4. 3 .2 )

• Keen and Morton (1978) provide a comprehensive review

of the evolving field of decision support systems. They

use a combination of case studies, system designs, important factors, and development models to illustrate the

design of these systems.

In addition, a wider range of variables should be considered. For

example, the elements of contracting and project management prevalent in many hardware system literatures should be integrated into

information system models that now exist. Techniques and lessons

for information systems might also be further supported by examples

from other kinds of systems.
147

The approach suggested above is used in this dissertation.

First, the four main approaches, case study, modeling, system design, and factor study, are used in combination. The monitoring

framework being developed for managing a project contains both design and process perspectives. Each of these areas contains a

range of factors found to be important in the literature. Several

case studies are used to illustrate the factors and their relationships. One special case study is used to trace these factors over

time. Secondly, factors for the framework are based on the important issues found across the diverse systems literature. The different perspectives available in hardware and software developments

of both public and private sectors are integrated into the framework

and used on the information system developments of central interest

in this dissertation.

3.4.3 Time Varying Qualities of System Development

Another perspective often lacking in the literature is the

relationship of time to development activities. It is difficult

to find sources which discuss how developments evolve over time

and are changed to accommodate varying situations. The treatment

of time also differs across the various kinds of literature.

The following sections review the dynamics of development as

observed in the literature. First, the different treatments are

discussed in each of the sectors used in previous sections: case
148

studies of developments, development models, important factors in

development, and system design. A second section presents the dynamic perspective rarely found in these sectors and how it will be

treated in this dissertation.

3 . 4. 3.1 Time Perspectives in the Literature

Case studies of system developments are one of the few sources

of information which demonstrate how developments proceed over time

and how changes occur in designs and strategy. The level of detail

can vary considerably however. Many case descriptions are brief

and focus only on the initial and final stages of a development.

Intervening details between these points which would illustrate

the evolution of strategy, problems, and changes are not included.

It is more difficult to find the cases which provide the comprehensive review and allow the reader to trace all stages of development.

One example of the comprehensive case is provided by Lucas

and Plimpton (1972) in their review of a development for the United

Farm Workers (UFW) (see section 3*3. 1.2 for a detailed description

of this case). The authors describe the model of development they

used and the implementation of the project. They include factors

such as how they identified information needs, contact with users,

problems with top management support, and deviations from their

model. This level of disclosure, although brief and lacking many

details, is rare in the literature. Readers are provided the opportu-
149

nity to observe the problems and uncertainties which characterize

a development and how developers compensated their strategy accordingly.

System development models are another sector of literature

where some amount of time perspective is available, although it

is less detailed than that available in case studies. Models provide a time perspective by outlining a sequence of project stages

which characterize changes in design and strategy. Most models

have stages such as problem definition, preliminary design, detailed

design, transfer, and full operation. While these are valuable

in providing an overview of development, they have limitations or

underlying assumptions which detach them from an actual implementation. For example:

• Models can have a step-by-step linear appearance.

• Models can give the impression that once the sequence

is initiated, it is always continued logically through

each step until project conclusion.

• Models can assume that all things outside the project

are held constant, such as resources or objectives.

These characteristics can easily misrepresent the delays, recycling,

or overlapping and parallel activities developments commonly experience. It is difficult to represent these conditions in a model,

however, unless greater detail can be provided through an accompanying case study or numerous caveats. Models usually are presented

at a general level without these supporting descriptions.
A time perspective is even less common in literature devoted

to describing important factors in system development. Factors

such as top level support or problem identification are usually

selected for specific emphasis in these sources. They are commonly

given a static quality in that their description may be at a general

level without reference to actual implementation or they may be

discussed within a narrow time slice of development. It is thus

difficult to gain a perspective on their importance throughout a

project and their relationship to other proceeding or succeeding

factors. For example, almost everyone agrees that user involvement

is a key consideration in any system development. The question

is what exactly this means during a project:

• User involvement could be extensive in some tasks and

not others.

• Early project stages might be characterized by the involve

ment of high level managers, while later stages would

mostly involve the operational levels.

• Users might be in charge of project teams and/or be partic

pants

• User involvement overall might not increase with time.

In addition, the implications or effects of various levels of user

involvement may be unclear. This is because involvement may not

always be tied to subsequent events, such as an accelerated development, expanding user involvement, or an improved design. Relationships like these must often be assumed in the factor literature.
Finally, time varying qualities are rarely of concern in the

system design literature. These sources usually focus on specific

design considerations or designs without reference to the needed

development activities. It is difficult to assume the development

requirements of a design unless the sources also include the background on how the results were obtained.

3 . 4. 3 . 2 A Dynamic Perspective Needed

From a developer’s perspective, it seems most useful to consider system development as a dynamic process (Eveland et al., 1977).

Designs evolve from a process where there are changing objectives,

alterations in the user organization and its environment, and changes

in the development process itself (Arnovick and Gee, 1978, p. 371).

The design and the organization will affect each other and cause

changes in each other. Decisions about a design will be interrelated over time and affect each other (Eveland et al., 1977, p. 64).

The systems literature provides pieces of this viewpoint, but it

is rare to find them combined in one source.

One modest example of the dynamic nature of development is

illustrated in an article by Alter and Ginzberg (1978). The authors

view system development as a changing process where uncertainties

and problems vary over time. These conditions add risk to the development at the stage in which they occur and can, if not resolved

at this point, eventually threaten full implementation of the sys-
152

tem.

Using the Kolb and Frohman model of development (see section

3 . 3 . 3*3 for a description), Alter and Ginzberg correlate risk factors with stages of the model. For example, they find two risk

factors especially significant in the diagnosis stage. According

to their model, diagnosis is the stage at which user and designer

refine the information needs and establish the basic concepts of

a design to meet them. Two factors, turnover and support for the

system, are critical during this time. Turnover is critical because

new participants at this point will likely cause changes in the

agreements reached or cause the project to repeat earlier stages

where initial work was performed. Similarly, lack of support for

the system becomes critical at this stage since users and designers

are attempting to specify more completely the design and move on

to subsequent development. The authors suggest strategies which

might be used to inhibit these problems in advance, or to compensate

for them when they occur.

While this article has its limitations, particularly in the

detail of the model and the lack of any supporting cases, it does

illustrate the dynamic nature of development usually lacking in

other sources. The authors emphasize the changing nature of a project and the need to change strategy. They attempt to pinpoint

critical problems in a development and locate them along a time

dimension. Development is characterized as a dynamic process which
153

must involve dynamic strategies from system designers.

This dissertation attempts to include this time perspective in

several ways so that the dynamic nature of development is more adequately represented. First, factors which can be used to measure

and indicate the evolving process are designed. In addition, factors

for measuring the concurrent aspects of system design and user commitment are also designed. By comparing process indicators to themselves

over time and to design and commitment characteristics, the dynamics

of process are demonstrated. Secondly, several actual cases are included to illustrate the factors. One set of cases is used to describe the factors (see Chapters 2 and 5), and one case is used to

test the factors (see Chapters 6 and 7). These cases provide more

detail on the difficulties and changing nature of developments than

is usually found in the literature.

3.4.4 User Commitment During Development

A critical aspect of a system development is the process by

which users decide to support, accept, and eventually fully implement a system. This process is not often well described or detailed

in the various areas of systems literature. The developer interested

in learning about the characteristics of the decision process will

find it difficult to assimilate an overall view for use in his own

situation
154

This section reviews the perspectives on the user decision processes available in the literature and then describes an alternative

model which developers can use. A note should be made about terms

at this point so that the reader can understand the background to

this section. The user actions or behaviors which are part of the

process by which a system is accepted into an organization is described

as user commitment in the following sections. Other terms could have

also been used, such as adoption (Flaherty, 1980, p. 65), implementation (Weiss and Thompson, Note 8), use (Lucas, 1975a, pp. 910-913),

acceptance, or even system "success" (Ein-Dor and Segev, 1978,

p. 1065). These are all used in the literature and would appear to be

almost interchangeable (Thompson, 1978b, p. 385). Commitment was

selected however since it connoted more of a quality of process:

it could evolve and build over time. It also emphasizes action: user

actions which bind the organization to future use of the system.

Conmitment is not an unknown term in the literature; many authors

refer to it with the general meaning intended here (Alter and Ginzberg,

1978; Brown and Watson, 1977; Comptroller General, 1975, pp . 7-8,

16-18; Lucas, 1978b; Miller and Wilier, 1977). However, the use

of the term is often confusing; its true meaning must be implied

from how literature sources use it in describing user behaviors.

Many imply that commitment is a point decision, essentially the final

decision to implement a system within the organization. Others use

the term more broadly as a factor which is operational during all

phases of development and at the final point where users must decide
about full operation. The general meaning of the latter approach

is the one which is of interest here. The following sections and

chapters use' the term in this sense and add more detail for its operationalization .

3 . 4.^.1 Commitment Perspectives in the Literature

An area which is poorly covered in the systems literature is

that of how organizations decide to support and eventually accept

a new system. Developers interested in this commitment process will

often find limited detail beyond brief discussion of factors like

the adoption or implementation of a system. For example, case studies

of system developments often exclude details on the middle stages

where the major design work is performed and user acceptance built

(e.g., Fudge and Lodish (1977), see section 3.3*1.1). These cases

instead concentrate on the initiation of the project and the resulting

successful adoption. Case studies which include the intervening details on user commitment decisions are not very common. A modest

example by Lucas and Plimpton (1972) demonstrates how users begin

building their commitment to a system and how this is subsequently

affected by later events (see section 3 -3. 1.2).

Other sectors of the literature provide differing, but similarly

limited perspectives on user commitment. Development models specify

stages and tasks for the different time periods of a project and generally include a decision point at the end of each stage. At these
156

points, users are to acknowledge the accomplishments of the previous

stage and set the directions for the next stage. This is a limited

view of user commitment since it focuses entirely on managers and

the few major decisions they make in a development. Other people

and events are involved which both precede and succeed these decisions.

Factor studies in the literature often hint at user commitment

actions missing from these models. For example, user commitment may

be implied in such commonly discussed factors as top level management support, user involvement, user satisfaction or attitudes towards

the system, or resource support to project teams. These factors may

be relevant at any phase of a development.

It is more common however for factor study sources to discuss

commitment type actions occurring at the end of a project. A distinctive, point in time, perspective may be taken, such as ’’the” decision to implement or adopt a system. Others discuss a more nebulous

"use” or "success" factor. Both of these approaches are then treated

as dependent variables which are influenced by independent variables

occurring earlier in the development, such as user involvement or

the degree of user training. These relationships create an unrealistic view of development in that causal linkages are established over

great lengths of time. An expansion of this approach into other parts

of development and with narrower portions of time would provide a

more relevant model of interaction.
System design literature is perhaps the weakest sector for learning about user commitment. Most discussions leave the impression

that only certain designs will achieve full commitment from users.

Very little evidence is presented to demonstrate how the results were

achieved, or that they were achieved over time.

Generally, few sources take the view that organizational decisionmaking on a new system is a process intertwined with other events

in a development. This would appear to be a serious weakness in the

literature since it is high ly unlikely that users decide to accept

a new, complex system in one grand deliberation at the end of a development. A sequence of events leading to eventual full implementation seems more realistic.

Some literature sources iupport this view. One illustration

is provided by Eveland et al. (1977) in their study of 53 system developments. The authors found that clear "adoption decisions" were

missing from the development phases. Instead, there were sequences

of decisions (either directly observable or implicit in actions),

each further defining the system and usually becoming more narrow

in scope over time. Others have also taken this perspective (e.g.,

Brown and Watson, 1977), emphasizing especially this sequence of broad

to more detailed commitment decisions.
158

3. ^.4.2 A More Comprehensive View of User Commitment

It seems most realistic in a complex development to consider

user commitment to a system as a process which begins at the earliest stages of development and continues throughout the life of the

system. The stages of the system life cycle concerning development

should be characterized by an increasing user commitment which builds

to a point where users are then ready to accept full responsibility

for further support and operation of the system. This process is

intertwined with the events of development. A cyclic interaction

occurs: development events influence user commitment and user commitment influences development (Lucas, 1974a).

Two additional improvements to this approach are needed to make

it comprehensive. First, the process should be considered as consisting of many different factors, direct or indirect. This recognizes

that varying kinds of events can indicate user commitment, and not

just the major decisions user management may make during development

stages. It is also a recognition that commitment is still an elusive

concept which is imperfectly represented by any number of specific

user behaviors. Commitment should be represented by both direct actions, such as documen table use, and indirect actions, such as changes

in the organization structure or functioning, which can be linked

to system requirements.
Commitment should be further expanded to include user actions

at the various different levels of the organization which are relevant in the development (Miller and Wilier, 1977, p. 205). All of

these have some relevance to an evolving user commitment and should

be considered with more equal importance. For example, managers must

devote resources to the system in order to support it, while operational levels must be willing to collect data to store for use in

the system. Each of these would be a significant indicator for user

commitment to a system.

The comprehensive view of commitment described here is used in

this dissertation as part of a mechanism developers can use to monitor

a project. User commitment is examined in several ways. First, a

number of factors are identified which can be used to measure a range

of user actions relating to commitment. Second, these factors are

illustrated with examples from several retrospective case studies.

Third, the factors also are used to explore commitment in a new case.

By comparing user commitment factors to design and development factors, this new case helps to demonstrate the evolving nature of commitment and its interaction with other factors in system design or development processes.

3.4.5 Calls for Research in the Systems Literature

There are numerous research needs in the systems literature

and they reflect both the breadth and depth of considerations potentially applicable in any system development, ’/^hile many of them
are relevant to this dissertation, several are particularly noteworthy since they relate to the general objectives and approaches

of interest here. These are:

• The need for more research on the management of system

developments

• The need for mechanisms to help developers find and apply

relevant systems literature.

• The need for more empirical evidence on the process of

developing systems.

Each of these needs is discussed below. A final section summarizes

how they have been included in the approach to the dissertation.

3. ^.5.1 Research Needed on the Management of Developments

The major need in the literature seems to be to increase the

amount of attention devoted to the management problems of system

development. Some authors believe that the literature, particularly

that related to behavioral, management, and computer sciences, has

been concentrated mostly on the technical aspects of development

(Kay, 1969; Lucas, 1974b; Lucas, 1975b). By this they mean that

many sources only focus on the mechanics of design: developing

technical specifications, building hardware or software, testing

designs, developing training programs, etc. They see a need for

more attention on coordinating the many people involved in a development, setting objectives, writing and revising project plans, or

reviewing and controlling progress. Better management procedures
161

in these areas is seen as improving the capability of developers

to deal with the complex technical problems (Smith, 1972, p. 129).

While there appears to be some basis for these assertions,

the problem may be overstated. Many of the factors commonly discussed in the literature already have either a direct or indirect

relationship to these management concerns. For example, in a study

on the management problems of information systems development,

Sollenberger (1971) indicates that one major problem is the continuity

of management. He concludes that continuity is a critical element

for coordination, control, and vitality in a development project

(Sollenberger, 1971, p. 11). Continuity is really another way of defining the problem of turnover that many other sources commonly recognize. Turnover is usually more closely associated with system users or

owners, however, then it is with project managers.

In addition, some sectors of the literature are weaker on project management than others. Thus the perspective on the needs for

more management research may be limited by a particular author’s

breadth of knowledge. This is easily demonstrated when comparing

systems engineering or defense related systems literatures with

others. Project management has long been a concern in these literatures and numerous techniques have arisen (e.g.. Air Force Systems Command, 1966; Chestnut, 1967; Hill, 1970; Peck and Scherer, 1962;

Peterson, 1980). Other models have also been developed in the public

sector, such as for the A-109 policy on systems acquisition (Office
162

of Federal Procurement Policy, 1976). All of these sources recognize the separate processes of design and the management of design.

Even a casual glance at the voluminous computer science literature related to information systems demonstrates the existence of

many detailed models and ideas for the management and control of

developments (e.g., Blumenthal, 1969; Davis, 197^; Hartman, Matthes,

and Proeme, 1968; Jones and McLean, 1970; Murdick and Ross, 1971;

Sollenberger , 1971). Some of the authors in this field may have also

benefited from the lessons available in the defense systems literature

(e.g.. Smith, 1972).

Arguments about the lack of guidance and research on the management of system developments are not then completely justified.

The problem can be better characterized by several other perspectives:

• Little recognition in some sectors of the literature of

the complexities of managing large scale development projects.

• Little transfer of knowledge across the various sectors

of the systems literature.

• Few methods or approaches to management in some sectors

of the literature.

These problems appear to be especially acute in the management and

behavioral science literatures. Some authors are beginning to recognize these gaps and are striving to generate new frameworks which

developers can use to manage projects.
163

3. 4. 5. 2 The Need for Management Frameworks

As demonstrated throughout this chapter, there are numerous

considerations prospective system developers need to make in planning and implementing a development project. The systems literature

is very broad and discussions on these considerations can be found

in the many different sectors of the literature. Some authors,

particularly in the behavioral and management science literatures,

feel that practitioners who develop systems are having a difficult

time making use of all the literature (e.g., Anderson and Narasimhan,

1979, pp. 512-513; Bostrom, 1978, p. 164; Lorsch, 1979). Some

even believe that researchers are having problems organizing the

extensive information already available and describing what is known

(Bostrom, 1978, p. l64; Ein-Dor and Segev, 1978, p. 1064).

This has resulted in a number of new frameworks or calls for

frameworks which in some manner can organize the important management problems of a development and offer the guidance already available in the literature. These frameworks cover a wide range of

variables and in most cases they reflect the specific perspectives

of the literature sector they are a part of. For example:

• Ein-Dor and Segev (1978) discuss user organization variables as they impact on the development of management

information systems.

• Lucas (1975b) examines user behaviors and attitudes.

• Kay (1969) and Smith (1972) outline general project organizations .
164

• Alter and Ginzberg (1978) and Anderson and Narasimhan

(1979) structure a framework of project implementation

risk factors.

• Bostrom (1978) develops a metaframework to categorize

many other frameworks from management and behavioral science

literatures.

The basic theme behind these emerging structures is that each

development situation is unique; project implementations must be

tailored to meet these circumstances (Williams, Note 9). System

developers need ways to diagnose the problems in their specific

cases and locate guidance to help solve them (Lorsch, 1979; Williams,

Note 9). This is especially the case for large scale complex system

developments where on-going methods to meet evolving circumstances

are needed (Alter and Ginzberg, 1978; Jones and McLean, 1970, p. 7;

Kay, 1969, pp. 425-426).

Existing frameworks have begun to fulfill this need, but more

comprehensive approaches must still be attempted. The models and

techniques of development available elsewhere for large hardware

or software projects offer guidance on project management that remains isolated from the perspectives of the behavioral and management

science literatures. Future frameworks need to combine the various

approaches and make them all accessible to prospective system developers .
3. 4. 5. 3 A Need for Empirical Evidence

In addition to frameworks for guiding managers of developments,

there is also a need for more empirical work to demonstrate how

management problems are handled in actual situations (Lucas, 1976;

Williams, Note 9). Management methods and ideas are rarely followed

by evidence from real developments which identifies why these methods

are important or what happens when they are used. For example,

Kraemer and King (1978) find these problems in their study of urban

information systems programs initiated by the Federal government

• Failure to provide for learning from the past and integration of these lessons.

• Little transfer of knowledge between applications.

• No documentation useful to other managers.

Part of the problem is that research on system developments

is difficult to perform. It is rarely, if ever, possible to conduct

structured experiments with developments where various factors,

such as different management techniques, can be tested or compared

(Brown and Watson, 1977; Resnikoff, 1979). System projects can

be large and uncontrollable; system developers may not be able to

integrate a research approach into their projects. A related part

of the problem may be that system developers have little incentive

to document their experiences for others.
However, real applications are the only source for empirical

evidence that others need. The best alternative for providing the

evidence would be to develop a research structure prior to the implementation of a development so that the important aspects of management could be documented as they occur. Generally, this kind of

research is difficult to locate. Case studies of actual projects

are usually retrospective in nature and focus on many other issues

besides project management. In addition, there are not that many

case studies available. Nevertheless, these studies are valuable

since they provide one of the few sources where readers can observe

the inner workings of a project. Until researchers can become a

part of the planning and execution of system developments, these

retrospective investigations are the best source for empirical evidence on management methods and more need to be undertaken.

3. 4. 5. 4 An Approach to Meeting These Needs

The approach used in this dissertation has been structured

partially in response to these research needs. First, a specific

facet of project management, monitoring progress, has been selected

as the central problem of this research. The main idea has been

to develop a procedure which can help developers gauge progress

in the project, identify important problems early, and enable managers to revise strategy in light of evolving circumstances. Second,

the factors selected for use in the monitoring framework have

been identified and examined across a broad range of the systems
literature. It is hoped that this coverage can demonstrate the

range of knowledge already available and the applicability of the

different perspectives to common, recurring management problems.

Finally, a number of case studies are used to support the development and testing of the framework. These cases are intended to

provide evidence of management problems at a level not commonly

found in available systems literature.

3.^.6 Summary and Conclusion

The previous sections have identified key development problems

and major research needs found in the systems development literature. Three major problems have been discussed. The first of these

is the problem of obtaining a comprehensive view of the many considerations potentially applicable in a system development. The litera

ture is segmented in certain ways which often lead to a high degree

of specialization in the presentations. For example, literature

sources can be divided by the range of variables considered in specific sources or by the specific systems (hardware or software)

of interest. A more comprehensive approach is needed which interrelates the major concerns in the various sectors.

A second problem with the development literature is the lack

of sources which demonstrate the time varying qualities of a complex

systems project. Many sources concentrate on specific factors impor

tant in a development without identifying how they are affected over
time by evolving circumstances and other factors. Models for system

development often leave the impression of a step-by-step sequence

to development which misrepresents the conmonly occurring delays,

recycling, and parallel nature of development activities. A more

dynamic perspective is needed which relates development factors

to a time dimension and demonstrates the changing, intertwined events

and problems of a systems project.

A third problem with the literature is the lack of sources

which describe the process by which users decide to support, accept,

and eventually fully implement a system. This process, defined

as user commitment in this dissertation, is intertwined with the

activities of development and evolves over time. Many sources identify commitment as an end event of a project which is influenced

by earlier factors such as user involvement or problem identification. Other sources which outline models for system development

identify only a few major decisions in each stage where users are

able to acknowledge progress and plan for follow-on strategy. A

more comprehensive view of commitment needs to be integrated with

these perspectives. This view should recognize the direct and indirect indicators of evolving acceptance and the user behaviors which

occur at different organizational levels.

A final section reviewed some of the major research needs which

are specifically identified in the literature. Three major needs

were discussed. First, more research is needed on the management
of system developments. Some authors believe that the literature

has been concentrated too heavily on the technical aspects of design

while avoiding the management problems relating to the coordination

and control of development activities. This seems only partially

accurate however, since some sectors of the literature devote more

attention to management than others. The problem is better characterized by three perspectives: the lack of recognition in some sectors of the management complexities of development, the lack of

transfer of knowledge between the various sectors, and the lack

of methods or approaches in some sectors of the literature.

A second need is for frameworks which outline the management

issues in development and provide guidance to practitioners based

on the already existing, voluminous discussions available in the

systems literature. This need is most recognized in the behavioral

and management science literatures which are slowly beginning to

identify the problems associated with the management of developments.

These sources are finding that each development situation is unique

and that developers need ways to diagnose the problems of their

specific cases and locate guidance to help solve them. Existing

frameworks in these sectors tend to focus on the specific aspects

of their tradition, such as the behavioral science focus on user

behavior and attitudes. More frameworks are needed which combine

these perspectives with the experiences and lessons of other literature sectors that have devoted more attention 1:0 project management

issues.
170

A final research need identified in the literature is the need

for more empirical work which demonstrates how management problems

are handled in actual situations. Sources presenting important

factors or models for managing developments are not often complemented by others which attempt to operationalize these concepts

and study their implementation throughout a project. Part of the

problem arises from the difficulty of structuring research experiments to study these factors. Part of the problem may also be the

lack of incentive developers have to report on their actual experiences. Real applications are the only feasible source to study management issues, however, and until research objectives are integrated

with on-going occurring system developments, more attention is needed on documenting and studying development cases in a retrospective

mode

Finally, the approaches used in the dissertation in response

to these problems and needs have been outlined in each section.

The goal of the dissertation is to develop and test a framework

developers can use to monitor system developments. The framework

attempts to provide a comprehensive view of the many considerations

important in a development and is based on the wide ranging perspectives and experiences available in the literature. It attempts

to recognize the dynamic, changing nature of development by providing an on-going means for developers to identify problems and revise

strategy in a structured manner. The need for more regular and

routine attention to user commitment and its interrelationship to
171

design and strategy has also been included as part of the monitoring

framework design. It is hoped that the development and testing

of the framework provides a basis for more research on the management of developments and offers an approach to the accumulation

of empirical evidence that can be used to illustrate developments

to others and organize further research in the systems literature.

3 . 5 CONCLUSION

This chapter has reviewed the systems literature for perspectives

and problems related to system development. Four main perspectives

towards development have been identified: case studies of system developments, important factors in system development, models for system

development, and system designs. Each area has been briefly described

and specific examples selected to illustrate the kind of material currently available.

The second half of the chapter has reviewed key development problems and research needs found in the systems literature. Three major

development problems have been identified after reviewing the literature: the lack of comprehensive view of development that integrates

the various considerations potentially applicable in a systems project,

the need for a more dynamic perspective of the evolving, changing nature

of development, and the need for a more comprehensive view of the process by which users decide to support, accept, and eventually implement

a system. In addition, three major research needs have been identified
172

by sources in the literature: the need for more research on the management of system development projects, the need for frameworks which can

help developers diagnose the management problems of their projects and

find the guidance available in the literature, and the need for more

empirical evidence which demonstrates how management techniques are

used in actual situations.

Some sections and paragraphs have been included in the problem

identification areas that reference the objectives and approaches used

in this dissertation. The purpose of including these references has

been to establish the linkages between the dissertation research and

already existing materials in the literature. Readers can follow the

continuing development of these linkages by examining the next chapters

on the research design and the monitoring framework design.

The next two chapters describe the objectives, approach, and the

major outcome of the dissertation. Chapter 4 outlines how the research

came about and what specifically was done during the research. Chapter 5 describes a framework for monitoring system development that

is the primary result of the research. Both chapters reference the

materials in Chapter 3 for additional background support.
173

4.1 INTRODUCTION

CHAPTER 4

RESEARCH DESIGN

As described earlier in Chapter 1, there were two main objectives

for the research described in this dissertation. These were to:

1. Develop a framework for monitoring system developments that

would provide a basis for assessing and revising strategy,

and to

2. Explore the usefulness of the framework as a management tool

in actual system projects.

Most generally, the research approach used to accomplish these objectives has been exploratory — there was significant uncertainty as

to methods, data, and results. As a result, the overall research design of the dissertation has evolved slowly over time as more was

learned about the management problems of system development and about

areas where a research contribution could be made.

This chapter summarizes the overall research design which has resulted from the exploratory approach. There are two main sections, one

for each research objective. First, the development of the monitoring

framework is described. This process is divided into several sections,

which trace the research from the initial idea to construct a model of

system development through to the resulting design of a monitoring framework. An overview of the framework and its functions is provided in

this chapter while more detailed discussions of these results are
174

presented separately in Chapter 5.

The second half of the chapter describes the efforts taken to apply

the framework to actual system development cases and demonstrate its

usefulness in project management. First, the initial effort to apply

the framework to on-going ETIP projects is described, followed by a discussion of the change to a retrospective approach. The design of the

retrospective application is then described in detail. The results

of the application are presented in Chapter 6.

4.2 DEVELOPMENT OF THE MONITORING FRAMEWORK

The monitoring framework described in this dissertation is the result of a lengthy study of systems literature, ETIP system projects,

and ETIP project management problems. The main ideas for the framework

evolved slowly from the interaction of these areas and there were numerous revisions of the monitoring problem.

This section describes the research process behind framework development. An attempt has been made to present the process as it

occurred, starting with the early ideas and ending with the monitoring

framework. However, the narrative has been divided into several sections in order to provide a better focus on the general direction of

activities during a given period.

The first two sections describe the initial perspective on the monitoring problem and the search for an ideal model of system development
that ETIP managers could use as a guide for project strategy.

The next two sections then describe how the research problem changed

to the development of a framework which managers could use to monitor

the key areas of an evolving system project. The refinement of factors

from the initial modeling effort is also described.

The fifth section reviews some of the literature that was consulted

during revision of the research problem. Perspectives in the literature

on the use of monitoring in system projects are described.

The sixth section presents an overview of the resulting monitoring

framework. First, the three main categories of monitoring factors are

briefly described. A discussion then follows on the five functions the

framework can have in project management and research. A flow diagram

is included to show how the framework can be implemented in a system

project.

The seventh section provides a brief guide to the dissertation chapters which resulted from the backgrounding and development effort.

4.2.1 Identifying the Problem Area

The ideas for the dissertation began during a new system development

undertaken by ETIP with several of its partner agencies. The purpose

of the project was to develop an evaluation system that agencies could
176

use to evaluate a series of experiments to be conducted with their policies or procedures. The project was to take three years, after which

a turnkey system would be institutionalized within the agencies. Two

contractors were hired by ETIP to help conduct the development work.

(This project was the original target for the monitoring framework.

The test application of the framework to this case is outlined in section

4.3 and fully reported in Chapters 6 and 7).

The development had been underway for a number of months when ETIP

managers (the author was part of the management team) began to see a

need for a way to improve the control and direction of the project.

The type of system being developed was new for ETIP and a number of complex problems were arising for managers:

• ETIP was an outside third party to the development, primarily

responsible for funding the project and managing the contractors. The contractors conducted the majority of development

work and the ultimate users were agencies not related to ETIP.

• Working relationships had to be established among the two con- tractors, ETIP, and several agencies. The development depended

on a close relationship between the contractors and the agencies. While this relationship had to be independent from ETIP,

ETIP still had to manage the contractor's work.

• The design and ultimate home for the system were uncertain

in the beginning, creating a level of uncertainty which had

to be managed carefully in both the partner agencies and the

contractors

• Numerous short term, start-up, problems were arising as the

working relationships developed. The partner agencies were anxious for early results and the contractors were searching

for more specific definition of the products they were responsible for developing.
• Attention to short term problems was distracting project managers from developing longer range strategy. Managers needed

a way to anticipate more of these problems and avoid a lengthy

series of high pressure, short term problem solving episodes.

• ETIP had difficulty relating on-going problems and progress

to overall strategy and making needed revisions in the strategy.

Initially, the general problem was considered to be one of strategy:

how to run the project to assure eventual success. A model for strategy

was needed to

• Help identify in advance the typical problems that should be

expected.

• Determine if the project was on course.

• Tie together short and long term perspectives during problem

solving.

A study was initiated by the writer to identify the strategy guidance available in other similar ETIP projects. ETIP had already conducted a number of system developments with other agency partners, although

these were not quite the same kind of system or necessarily of the same

magnitude as the proposed evaluation system. Several of these cases

were selected for in-depth study with the main idea of identifying the

development approach of the project and tying the approach to the degree

of success of the system. Factors inhibiting or facilitating the project

were noted and attempts were made to trace the factors over time and

determine their relevance in different project stages. It was expected

that the variation in approach and success across the selected cases

would ultimately provide ETIP managers with a model of development

they could use for the new evaluation system.
178

>/hile a number of important factors concerning the development process were identified in these initial studies, the case study approach

eventually proved to be unsatisfactory. Tracing variables over time

required more information about the projects than was usually available.

It was difficult to reconstruct project events so that the interaction

of problems, strategy, and success could be identified and used to produce a model. Also, it appeared that the level of guidance available

from the few selected projects might be more limited than had been originally expected. This prompted a search of the systems literature for

additional ideas.

4.2.2 Reviewing the Literature and Selecting Important Development

Factors

A search of the systems literature was initiated after the ETIP

case studies in order to identify additional guidance for the desired

model of system development. Since the case studies had produced a number of important factors that managers should consider in a development,

it was decided to continue the factor approach. Lists of factors were

started in two areas, one for factors relating to the development process

and one for factors relating to project success. It was expected that

the expanded search would provide enough factors to then synthesize the

strategy model ETIP managers desired.

A broad review of the literature ensued as it became obvious that

a great amount of material was available from numerous perspectives and
disciplines. Much emphasis was initially placed on the behavioral and

management science literatures as these were of primary interest and

familiarity to the writer. Later, the similarity of problems and ideas

from different types of sources demonstrated that it would be useful

to include literature from a range of backgrounds and systems. Thus

the search was expanded to areas like systems engineering, weapon system

acquisition, operations research, computer system development, and other

miscellaneous hardware and software system literatures. These areas

were scanned and probed briefly; an in-depth search of each area was

considered unnecessary since the primary objective was to identify factors, not cover the literature (Frederick, Note 1).

Several guidelines were used to select factors for the development

model as the literature was examined. The first guideline was to select

factors which appeared relevant to the ETIP management and project environment, especially factors not previously identified through the ETIP case

studies. The needs of the new evaluation system project were also a

consideration under this guideline. Some important factors were already

evident in the project as the dissertation started and the literature

reinforced the need to include them in the model. Other factors were

drawn from the procedure used to procure contractor support for the project. The parametric factor evaluation (PFE) procedure developed by

Thompson (1976) for the procurement (later studied by Libman, 1980) contained a range of factors which were used to evaluate contractor proposals. Some of these were found to be useful indicators for on-going

monitoring as well (Thompson, Libman, and Garrity, Note 2). For example,
180

one key factor was the importance of the project to the bidder. Companies receiving a high score on this factor were considered to have a vital

interest in the project, particularly in relation to their company’s

objectives and top level management support. This factor was considered

equally important during the project; companies maintaining a vital

interest in the project would likely continue to exhibit top level management support. Continuing contractor interest was one factor essential

to project success, and thus it was included as an indicator to be monitored.

A second guideline was to select factors found to be common or important in the literature. For example, since user involvement was a factor

most everyone considered important, it was included in the factor list

for development process. However, factors did not necessarily have to

be widely recognized across literature sources in order to be listed.

For example, project management issues were considered important in defense related literature but rarely recognized elsewhere. They were

also included in the model. Generally, it was found that literature

sources could be very selective in the issues they emphasized, further

supporting the need to review a broad range of literature. This selectivity also highlighted a number of gaps in the literature which were

of importance to the model development. These gaps had an influence

on the dissertation by indicating areas were a research contribution

was needed (see section 3*4 for more discussion).
A final guideline in factor selection was the interest of the writer. In some instances, this involved noting factors not widely discussed

in the literature, as described above. In other areas, factors were

avoided which seemed to be tangential to the monitoring model being developed. For example, numerous models for defining decisionmaker needs

were found (e.g., Taggert and Tharp, Note 3) as essential components

to the development of an information system. Some of the essential characteristics of models like these were important to the monitoring model

being developed for ETIP, such as the need to have users involved early

in the definition of the information problem. However, the numerous

intricacies of the problem definition models were considered beyond the

detail required for the monitoring model. Other examples of uninteresting factors were organizational characteristics, such as organization

size, budget, or decisionmaking structure, which were commonly used as

key factors in system adoption studies. These factors appeared to fall

outside the themes of development process and project success that were

guiding the development of the strategy model.

It should also be noted that the lists being developed did not squally reflect all of the literature sectors. Sources in the defense related

systems literature and in private companies developing major systems

were not covered to the extent really needed. This was partially due

to the initial focus on behavioral and management science literatures

most familiar to the writer. Later, the literature search was curtailed

to avoid going beyond the scope of the dissertation effort. Additional

research covering the experiences in private companies and defense sys-
182

terns would have been useful, but had to be deferred until after completion of the dissertation.

4.2.3 Redefinition of the Monitoring Problem

A long list of variables, theories, and ideas on system development

emerged from the literature review. Approximately 70 items were listed

under the two categories of development process and project success.

The wider range of items in these lists compared to those first obtained

from the ETIP cases justified the long effort that had gone into searching the literature.

It was difficult, however, to assemble all of the items into a model

usable by ETIP managers in the newer, on-going evaluation system development. Several problems were arising as the lists were reviewed and refined for the model:

• No one model for development was uncovered in the literature

or emerging from analysis of the lists that seemed readily

applicable to the ETIP situation.

• Many items overlapped each other, requiring detailed examination to determine which items were more encompassing in meaning.

• Items ranged in content from a linked set of factors to only

one important factor. Synthesizing a model required refining

these items into a form where they could be compared or com- bined (see following section 4.2.4).

• Many of the factors listed under the development process category were found to be more focused on technical design problems

than on the operation of a project.

• Some factors were found to be very situation specific, making

it difficult to place them in a general model for strategy.
In addition, it had been difficult to find sources which recognized

the characteristics and problems of large scale projects of the kind

undertaken by ETIP. Few sources discussed the strategy aspects of a

project where significant, on-going change and uncertainty were components managers routinely had to face. Also missing were the comprehensive views of development which specified the broad range of factors

managers might have to consider in devising a project strategy. Thus,

models for strategy that ETIP could use were hard to find and difficult

to create; available models appeared to be lacking in one aspect or

another

It was at this point that comparison of the lists with ETIP management needs began to show that a different kind of monitoring structure

was needed. Initially, the structure had been expected to characterize

a model for development, i.e., a strategy to guide ETIP managers in the

conduct of the new evaluation system project. Based on strategies found

to be successful in other projects, this model would have provided a

basis for monitoring project activities and determining if the project

was on course.

However, the real underlying problem facing ETIP managers was one

of information, not strategy. ETIP managers needed better information

about project activities to identify when and where problems were occurring and whether elements of their strategy needed revision. The structure for monitoring in this case had to focus attention on key project

areas, areas likely to be sources of recurring problems and/or critical
to overall project success. For example, the monitoring structure needed

to contain a factor such as the structure of project tasks, advocating

the need for ETIP managers to review the sequence and priority of tasks

as the project unfolded. Included with the factor would be criteria

for identifying task structure problems and guidance on what considerations were needed to make changes in tasks. A monitoring structure or

framework of factors like this would be oriented more toward measuring

performance than specifying an approach that had to be followed by project managers.

Definition of the monitoring problem in this form caused several

changes in the background work and the design of the monitoring structure. First, the monitoring structure was renamed as a framework to

better represent the concept of monitoring a project on selected factors

rather than implying a model for project strategy. Second, much of the

material collected in the factor lists had to be refined to identify

the key, underlying elements implied in the items (discussed further

in the next section). Third, a category for design characteristics was

added to the framework, recognizing the importance of design related

factors in the success of system developments. The monitoring framework was thus designed to contain three categories of factors:

• Process characteristics.

Factors specifying methods, organization, personnel, and proce

dures that are essential elements of management and control

in a project.

• Design characteristics.

Factors specifying the design and functioning of the evolving

system that are key sources of technical problems during a development
• User commitment characteristics.

Factors specifying user actions related to the support and

acceptance of the system; early indicators of success in

institutionalizing the system in the user organization.

4.2.4 Refining the Framework Factors

One of the problems with identifying important factors for system

development was that many items listed in the initial search were very

general in scope or implied a number of underlying factors. This made

it difficult to combine these items into a strategy model as originally

intended; later, they were found to be too general for the redefined

monitoring framework where more basic project elements were needed.^

Multi-factored items were most prominent in the list of development process factors where numerous models or strategies for development had been accumulated. A simple example of one of these models was

the commonly discussed evolutionary strategy to system development (Davis

1974, pp. 405-409; Lucas, 1978b; Mohan and Bean, 1979). Evolutionary

in this usage meant the use of small, incremental steps in the process

of designing the system: working on small pieces of the problem, develop

ing subsystems individually and in series, changing over to a new system

slowly, and minimizing the disruption to the user organization. This

strategy was supposed to assure project success by keeping change small

and achieving a series of modest successes.

Initially, it had been expected that items like this would be used

to formulate a model strategy well suited to the ETIP situation. Thus

An initial test of the framework on the ETIP evaluation project

at this point also demonstrated problems with the factors. See

section 4.3.3.
186

elements of the evolutionary strategy would have been combined with other

factors to produce a new model, similar in content to the stages and

activities implied by models like the evolutionary strategy. However,

revision of the monitoring problem required that these models be further

decomposed into their basic elements. These elements had to identify

key areas for monitoring the performance of strategy, rather than specifying the strategy required. An example of another model illustrates

what had to be done for this approach.

In an article by Mumford et al. (1978), several models were found

for the design of data processing systems. One model, outlined in

Figure 4.1, described a small chain of events that characterized

what some authors considered as the conventional, technical approach

to system design. The model showed how a design group which is oriented

more toward the technical aspects of a development can lead to a system which produces high user anxiety and dissatisfaction. Essentially,

the authors were implying a number of factors about system development

that could be divided into design, process, and user commitment categories. These were:

• The need to have a design team balanced with technical and

human engineering skills (process characteristic).

• The need for user involvement in design activities (process

characteristic)

• The need to have the design match the user environment (design

characteristic)

• User actions indicating decreasing acceptance of the system,

such as sabotage, non-use, or the lack of champions to defend

it (user commitment characteristics).
Model of Design Events Some Underlying Elements for the

Monitoring Framework

Project personnel should be

multi-talented; technical

skills should be complemented by behavioral skills.^

No user involvement.

Design not well matched

to the user.

Sabotage

Lack of system champions

Non-use.

FIGURE 4.1 DECOMPOSITION OF A DEVELOPMENT MODEL

Process

factor

Process

factor

Design

factor

User

commitment

factors

2 Adapted from Mumford et al. (1978).
188

These were the kind of elements that the monitoring framework needed.

Each model in the factor list had to be examined in the same way to identify the common core set of factors they implied.

Similar analysis was also needed for many of the items which specified only one factor. This was usually due to the general level at which

the item was written, often lacking any operational implications. For

example, user involvement was one factor often identified in the literature as essential to project success. However, few sources illustrated

the concept with actions for different project activities or stages.

Since users could have a number of different roles in the management

and support of a development, these actions had to be thought through

to establish what should be used to indicate user involvement. User

involvement also had to be linked with other factors in the design, process, and user commitment categories so that the effects of involvement

could be also monitored elsewhere.

4.2.5 Monitoring As Seen in the Literature

Redefinition of the monitoring problem made it necessary to return

to the systems literature to identify how the problem was already discussed. As noted earlier, few sources seemed to capture the complex

development situation ETIP faced; thus it was expected that locating

relevant sources with the needed view of the monitoring problem would

be difficult. However, the idea of monitoring a system development on
key factors to help identify problems and revise development strategy

was not examined in sources during the first literature search.

It was found that the general idea of monitoring a development was

recognized in a number of different sources (e.g., Alter and Ginzberg,

1978; Anderson and Narasimhan, 1979; Comptroller General, 1975, 1978;

Gordon, 1980; Keen and Morton, 1978, p. 204; Lucas, 1974b; Waller

et al., 1976). Some of these sources presented a limited view of monitoring, such as simply to periodically compare project status with original

plans (Comptroller General, 1978, p. 8; Lucas, 1974b, p. 136). Others

considered a more active, evolving role for monitoring, seeing it as

essential to ensuring project success (Alter and Ginzberg, 1978; Anderson

and Narasimhan, 1979, p. 513; Keen and Morton, 1978, p. 208). These

sources recognized that initial planning was important and could act

as benchmark for gauging later progress. However, they saw monitoring

as more than a simple before and after comparison:

• Active monitoring could provide early visibility to problems

and timely action to their solution (Gordon, 1980, p. 32;

Smith, 1972, pp. 129, 135; Waller et al., 1976, p. 9).

• Monitoring could provide a means of collecting project information to routinely keep developers informed of progress (Smith,

1972, p. 131).

• Monitoring could be used to guide revisions of strategy on

a continuous basis (Alter and Ginzberg, 1978; Anderson and

Narasimhan, 1979, p. 513).

• Monitoring could provide a means to better manage the problems

of isolation (between managers, developers, and users), uncertainty (in project design, system design), and volatility (instability of design, environment, or personnel) (Ibrahim, 1978,

pp. 34-36).
190

• Monitoring could be used to audit and evaluate a project

implementation (Keen and Morton, 1978, p. 204).

These views tended to correspond more to the concepts of monitoring that

were evolving in the dissertation.

Other views of monitoring were also identified from the researchers interested in the process of developing systems. Several researchers described a need for better organization of the literature so

as to provide a means for busy practitioners to access the guidance

in the literature (Anderson and Narasimhan, 1979, p. 513; Bostrom,

1978, p. 164). One approach suggested was to develop frameworks of

important development factors which practitioners could use to identify problems in their projects and then locate relevant literature

sources (see also section 3*4.5). While these proposed frameworks

were usually less comprehensive in scope than desired in the dissertation, they did approach the idea of a structure of key factors which

developers could use to examine their own situation on a regular basis.

Other sources described a need for more research on the management of system developments, particularly emphasizing the need for

more empirical evidence on the development process (Lucas, 1976;

Williams, Note 4). While these discussions did not explicitly mention

monitoring, they did highlight the idea that monitoring might have a

research role. Besides meeting the administrative needs of project

managers, monitoring might also provide a means for researchers to

collect data on developments in a structured, real-time manner. Re-
searchers could thus join with administrators and use the framework

as a channel through which they could explore or test concepts of

benefit to the literature.

Beyond suggesting the administrative and research benefits monitoring might provide, the literature was fairly lacking in structures that

might actually be used for monitoring. A comprehensive approach of the

kind needed by ETIP, including design, process, and user commitment

characteristics, was not available. Moreover, few sources really

recognized the kind of elements a monitoring framework should contain.

These gaps further supported the research underway in the dissertation.

4.2.6 The Resulting Monitoring Framework

The lengthy background studies finally resulted in a framework of

30 factors for monitoring system development along with some procedures

for how the framework could be used.

The following sections briefly describe these results. First, an

overview of the monitoring factors is presented. Then the five functions

of the framework are outlined. Further details on the factors and functions are presented in Chapter 5.

4. 2. 6.1 Monitoring Framework Factors

Thirty factors important in monitoring system developments have

been identified in the background studies (see Figure 4.2). The factors
192

FIGURE

H.2

THE

MONITORING

FRAMEWORK
are divided into three main categories as discussed earlier: design,

process, and user commitment characteristics. Several subcategories

have also been established.

Design characteristics are factors that developers can use to monitor

what a system looks like and how it will operate. Numerous literature

sources and ETIP experiences demonstrate that design problems can seriously affect development progress and thus should be a part of a system

managers use to monitor a project. Common problem areas in design are

factors such as response time or quality of output. Others areas less

commonly recognized relate to system boundaries or the adaptation of

the design to the user organization. Developers need to identify both

problems and progress in these areas to determine if project strategy

is working, if the evolving design is sound, and if the design is likely

to gain user support.

Process characteristics are factors that developers can use to monitor the means by which the system is developed. Factors related to the

development process are perhaps the most numerous in the literature;

the well recognized need for user involvement is one example. Other

important issues concern project management, such as staffing, reporting,

or defining priorities. Developers need to monitor these characteristics

to determine if they are matched to the needs of the evolving design

and are ensuring the appropriate level of user involvement and support.

User commitment characteristics are factors that developers can
194

use to monitor actions related to support for the evolving system. This

support is seen as evolving over time. There are both direct and indirect events which can be associated with user interest and acceptance

of the system design during each project stage. Developers need to monitor these actions to determine if support is increasing or decreasing,

and whether changes in the design or the development process are needed

given either of these conditions.

It should also be noted here that while the framework is intended

to be comprehensive (in that an attempt has been made to include the

major factors developers should consider), revisions and additions may

be necessary and are expected. It is likely that gaps still remain in

the framework elements and that there are alternative structures which

might be an improvement for each of three main categories. The need

for continuing revisions results from several circumstances:

• This is the initial attempt by the writer to define a comprehensive framework.

• The original intent was to identify as many important factors

as possible. However, it was not expected that the framework

would necessarily be completed within the limitations of the

dissertation

• The framework is oriented towards the information system developments at ETIP; other situations and systems may require revisions .

• Some literature sectors (especially the defense literature)

have not been covered to the extent needed. Better approaches

may be uncovered.

• Further revisions were expected during the test of the framework on the new ETIP evaluation system development (see Chapters 6 and 7 )
Framework development is thus at an interim stage and changes are

likely. Readers may find the current framework most helpful as a

guideline to be applied, with revisions, to their own specific cases.

4. 2. 6. 2 Using the Monitoring Framework

Overall, monitoring, as developed in this research, means both the

collection (on selected factors) and the use (for selected purposes)

of information on events associated with the project (Waller et al.,

1976, p. 5). As discovered during the background studies, monitoring

can potentially have two basic roles — one administrative and the other

for research. For administrators, a monitoring program using the framework can provide a means for better management control of a systems project. For researchers, the framework can also provide a means to conduct

structured investigations of system development projects. These two

purposes are seen as complementary. Research with the framework can

provide a basis to support and evaluate management actions; management

use of the framework can provide a test arena for research (Thompson

and Rath, 1974).

Within this dual purpose context, the framework has been accompanied

by a set of five functions, all of which overlap and support each other.

These functions include problem identification, strategy development,

research, documentation, and dissemination. Briefly, these functions

are described as follows:

• Problem identification — The tracking and assessment of key

areas where development problems typically occur, both in the
196

short and long range.

• Strategy development — The identification and formulation

of explicit management actions aimed at solving problems and

revising strategy.

• Research — The design and implementation of real-time studies

of the development process.

• Documentation — The implementation of an organized and stable

recording process which can identify special or recurring

problems, support decisionmaking, and research.

• Dissemination — The distribution of key development information to those inside and outside the project.

The use of these functions in a system development project is modeled

in Chapter 5.

4.2.7 A Guide to Chapters on Framework Development

Several chapters have been written to document the studies and results of the framework development effort. Chapter 2 describes the three

ETIP projects used to initially define the monitoring problem and later

to provide illustrations of framework elements. Chapter 2 briefly describes the history of each project and reviews some of the important

development factors uncovered in each case. Besides helping to illustrate parts of the monitoring framework, these case studies may help

familiarize the reader with the ETIP environment and the systems projects.

Chapter 3 reviews the literature used to help refine the monitoring

problem and identify more factors for the framework. The chapter is

divided into two main parts. First, the literature covered in backgrounding is generally described and some comments are provided comparing the

different sources found. Several sources are selected to illustrate
197

different sectors of the literature, and some of these are later used

to support the descriptions of framework factors in Chapter 5. The

second half of Chapter 3 describes several gaps in the literature which

have influenced framework development. Included in this discussion is

a brief review of calls for research that can be found in the systems

literature

Finally, Chapter 5 presents the monitoring framework and functions

which have resulted from the background studies. The chapter is divided

into two main parts. First, the framework factors are presented. Each

of the 30 factors is briefly defined, discussed in more detail, and then,

in most cases, illustrated with examples from some or all of the ETIP

cases discussed in Chapter 2. The second portion of the chapter presents

the five functions the framework can serve in a development. A general

model of how these functions might be implemented in a project is also

provided

4.3 EXPLORING THE USEFULNESS OF THE FRAMEWORK

4 . 3 . 1 . Introduction

The second objective of the dissertation research was to explore

the usefulness of the monitoring framework in actual system development

projects. This was to be accomplished through the application of the

framework to several on-going development cases in ETIP and evaluation

of the framework in helping to identify and solve project management
198

problems. However, the lengthy development period for the framework

altered the plans for these applications. Eventually, only one

retrospective application was conducted.

The next several sections summarize the initial attempts to apply

the framework in real-time and then describe in detail the design of

one retrospective case ultimately conducted.

4.3.2 Application to On-Going System Developments

As described above in the framework development presentation (section 4 . 2 ), the research project began during the development of an evaluation system in ETIP that involved the writer. The idea at the time was

to develop a model of system development that would prescribe an idealized approach or strategy based on the experiences in other ETIP projects

and the systems literature. After developing the model, it was to be

tested on several ETIP projects underway during the period, including the

evaluation system project. The different projects were expected to provide a range of different conditions for testing the model.

However, the form of the model changed considerably as further study

was made of the problem. Analysis of several completed ETIP projects,

the literature, and the on-going activities of the evaluation system

project showed that project managers really needed a means for monitoring

key problem areas as the project unfolded, rather than a model that specified a strategy. This realization significantly changed the direction

of the research underway at the time and further delayed any applications
to the existing ETIP projects.

Eventually, the development of the monitoring framework exceeded

the duration of the evaluation system project and precluded making a

real-time test. Consideration then began on the possibility of conducting a retrospective application of the framework to the same project.

Other additional applications to ETIP projects were also terminated at

this point since it appeared that the amount of effort required to include them exceeded the bounds of the dissertation.

4.3.3 An Initial Retrospective Test

Initially, it was unclear how to conduct the framework application

in the retrospective approach. The framework at that time simply consisted of a list of factors important for monitoring and the voluminous,

unorganized records for the evaluation system project. It was uncertain

what set of events was needed for the application and how events could

be used to demonstrate the utility of the framework.

It was decided to examine the first six months of the project and

to identify events that related to framework factors. This period was

selected because of the writer’s familiarity with it and the availability

of extensive records. The approach was to identify events in the various

sources and then classify them under the relevant framework factors.

For example, events which related to reports and reviews were listed
200

chronologically under that factor. This was expected to show that the

framework could be mapped onto the project, and specifically, that:

• Elements of the framework were relevant to the system development case.

• Events often involved or related to a number of different framework elements.

• Elements of the framework could be traced over time and that

the relationships between elements could change over time.

Documentation of these points was expected to demonstrate that the framework could be useful in monitoring and managing the complex problems

often present in system developments.

Several months of the project were examined using this approach,

but the application ultimately proved to be premature. It was found

that the initial state of the framework was simply too undefined to result in a consistent classification of project events. For example,

some elements of the framework overlapped considerably, making it unclear

where a particular event should be listed. It was also found that some

events were not easily classified under any of the elements. While these

problems were expected under any framework design, the difficulties in

this case were becoming too extensive for demonstrating that the framework could be mapped onto the project.

At this point, it was decided that further delay in the retrospective application would not cause any significant problems and that the

3 framework should be revised. Several changes were then initiated:

3 These revisions are discussed more fully in sections 4.2.4 and 4.2.5.
• Elements of the framework were reexamined and reorganized in

order to decrease overlap or specify new areas.

• Further study was made of how the framework could be used in

a project. Five functions were then identified: problem solving, strategy development, research, documentation, and dissemination .

• Framework elements were defined and illustrated with examples

from the three ETIP cases initially studied to identify important factors in system development.

These changes resulted in an improved framework and documentation

of the various materials used in its development and definition. These

included Chapter 2 (the three ETIP cases). Chapter 3 (the literature

background), parts of Chapter 4 (the framework development process),

and Chapter 5 (the detailed descriptions of framework elements).

The retrospective application was then reexamined to determine the

best approach for its accomplishment.

4.3.4 A Second Retrospective Application

4. 3. 4.1 Revising the Application Design

The initial approach to the retrospective application raised severa

other problems besides those associated with framework design. First,

it was obvious that relying on project records for the application was

going to leave gaps in events. In addition, project documents did not

always present the overall perspective of events that project managers

had during the periods of interest. These problems led to the question

of whether project records would be sufficient in themselves to conduct

a reasonably accurate application of the framework.
202

A second major problem was time. It was clear that an extensive

review of the project over its entire period would require much more time

than expected. This time would also be significantly increased if any

additional efforts were taken to supplement the project records, such

as in conducting surveys or interviews of project participants. An extensive amount of time had passed since the occurrence of project events

and any renewed contact with participants involved problems of recall

and reliability.

The third problem raised by the initial application was that the

simple classification of events appeared to add little more to showing

the usefulness of the framework than the examples already included in

the descriptions of framework factors. Mapping the framework onto a

new case such as the evaluation system project was considered useful

in further demonstrating the relevance of the factors, but the additional

illustrations seemed repetitious.

These problems prompted a revision in the design of the application

since it was felt that no application would leave a significant gap in

the dissertation research - both in demonstrating how the framework could

be used and why it should be used. It was decided to attempt several

smaller applications within the time period of the project rather than

attempting to cover the entire period as one case. This led to the idea

of working with selected periods of the project where there seemed to

be a dominant problem that the monitoring framework might have helped

project managers solve. An application would thus include descriptions
of the problem, identification of the related framework elements, and

analysis of how monitoring the elements might have altered the problem

and its associated events. This approach provided two ways to demonstrate the usefulness of the framework:

1. Mapping the framework onto the problem and its related events

would demonstrate the relevance of the framework to system

development activities.

2. Proposing alternative courses of action based on the mapping

would demonstrate how the framework as a whole could contribute

to improved project management.

The specific activities undertaken in this approach are described

in the following sections. First, the process used to develop a chronology of the project and help identify periods with a dominant problem

is presented. The framework application procedure is then described.

4. 3. 4. 2 Developing Chronology of the Case

The first task of the framework application was to develop a chronology of the ETIP evaluation system project. This process was to serve

several purposes:

• Outlining events in the project and familiarizing the writer

with the history.

• Identifying events or periods of time which might be useful

for illustrating the framework.

• Developing a brief description of the project to include in

the dissertation.

Work began by collecting all available project documents in ETIP.

The set of records obtained included the personal files of most ETIP
204

staff directly involved in the project (including the writer's), the

official project files, and the library of reports generated by the two

contractors

The documents were then filed by date and a listing of project events

was generated. It was found during this process that while the project

had lasted over five years, most of the system development activity had

4 occurred within the first three years of the project. Thus, it was

decided to concentrate the chronology on the first three years and only

briefly summarize subsequent periods.

It was also found that the first two years of the project were the

best documented. There was an extensive number of reports, memos, notes

on phone calls, and contact reports on meetings between various project

participants for this period. The third year was primarily represented

by several letters and final reports, leaving many gaps in events and

thinking. The search for framework applications was thus subsequently

focused on the first two years.

The final step of the chronology process was to write a summary

of the project for the dissertation. Since the purpose of the summary

was to help familiarize the reader with the basic project history, the

4 This was not unexpected since the project had been designed to last

three years, ending with institutionalization. However, the project

was extended several times after the third year and it was unclear

whether development had been continued.
summary was designed to be a brief and factual overview of major project

activities. 5 The following procedures were used:

• Emphasis was placed on events concerning project management,

reports, and system development. Discussion of the technical

evaluation activities in the project (i.e., events concerning

specific evaluation reports) was minimized.

• Interpretations by the writer were minimized by avoiding broad

summaries or comments as much as possible.

• The summary was divided into six month periods since the

project had been divided in this manner during planning.

• Citations to specific documents were not used since the descrip

tion was only to be a brief summary for readers. (Citations to

project documents were to be used for the framework application

however.

• Specific names of individuals were not used. In addition,

neither contractor was specifically identified.

It was decided to include a section on the background and objectives

of the project to help readers understand how the project was designed

and the environment in which the system was being developed. Material

for this section was adapted from the detailed background chronology

of events developed by Libman (1980).

The resulting chronology is presented in Chapter 6

4. 3. 4.

Applying the Framework

Development of the chronology helped identify several periods in

the evaluation system project where one problem tended to dominate and

affect other events. For example, during the first six months of the

5 The focus on major events and problems helped identify areas where

the framework could be applied (see next section).
206

project, July to December, 1976, a significant amount of attention was

given to the development of a system for reports and reviews in the project. While this amount of concern seemed justified given the communication needs in the project, the resulting system proved ineffective

and contributed to many communication problems. In addition, problems

with the system affected several other areas of the project. Overall,

this situation, along with several others, appeared to be excellent examples where the usefulness of the framework in project management might

be shown.

A procedure was then developed to analyze these periods of the project and apply the framework. First, the period of interest was studied

by tracing the events related to the dominant problem identified. For

example, in the case of the first six months, events related to reports

and reviews were collected in a list. This tracing provided the basic

data to establish the evolution of the problem, the specific components

of the problem, and the actions taken by project managers to resolve it.

Next, the same period was reviewed again to determine how the problem was tied to other events in the project. For example, the problems

of reports and reviews were examined to identify any effects they had

on events related to the design of the evaluation system. Areas of the

project which were affected by, or had an effect on, the dominant problem

were then classified according to the relevant framework elements in

any of the three main categories: system design, development process,

or user commitment. This procedure established the mapping of the frame-
work onto project events.

Finally, using the mapping created, alternative courses of action

were explored to identify how the actual sequence of events leading to

the problem might have been altered with the framewwork to avoid the

problem. The emphasis in the analysis was to show that:

• The framework might have helped identify the problem sooner.

• The framework might have helped decompose the complexity of

the problem and its effects on other project areas into simpler, manageable elements.

• The framework might have helped design improved alternative

courses of action.

Each of these separate studies were then documented as one application. Separate sections included evolution of the problem, specific

components of the problem, effects on other areas of the project, how

the problem was resolved in the project, and how the framework might

have helped resolve the problem in a different manner. Since these sections relied on details of events not specified in the chronology, specific citations to project documents were included in the discussions.

Three different periods were initially selected as separate cases

for a framework application. Each of these involved the dominant problem

desired for an example:

• July-December , 1976 — problems with reports and reviews.

• April - September, 1977 — problems with the first prototype

system design.

• March, 1978 - 1979 — problems with institutionalization of

the system.
208

However, only one application, reports and reviews, was completed

for the dissertation because of the time limits allowed for the research.

4. 3. 4.

Limitations and Biases of the Approach

The approach used in the retrospective application has its limitations and these should be noted by the reader as the application is re- viewed. First, relying primarily on existing ETIP records for project

information means that some events were probably not identified and that

the overall importance, relevance, and/or priority of events as viewed

by project managers at the time were not always known. Thus the selection and use of events in the application may be biased toward activities

that were documented or the significance of some events may be incorrect.

Second, the project records used in the application are generally

not available for readers to scrutinize. Many of the materials used

are notes and memos which are not a part of the official record; even

some of the official reports are not available outside of ETIP. Thus,

to some extent, readers cannot easily verify that the events selected

for the application actually occurred or that the usefulness of the framework is plausible given what actually happened in the project. The bias

of the application is thus difficult to check.

Another source of bias is the writer, first from being involved

with the project. The writer has special knowledge of the evaluation

system project due to participation in the ETIP management team for the
209

system project due to participation in the ETIP management team for the

first l8 months. While this was an asset to the extent that it facilitated use of project records and supplemented them with an insider's

perspective, it also established a particular viewpoint on events of

the project. This viewpoint does not necessarily correspond to what

other participants, such as the COTR, had during the project or would

have now. Thus, all aspects of the retrospective application are from

the writer's viewpoint, which is not necessarily complete nor totally

accurate

A second writer bias involves the selection and interpretation of

events in the project record. The extensive records of the project have

been carefully reviewed to identify a dominant problem for framework

application, map the framework onto events, and suggest ways that the

framework might have altered events. The events selected to document

the application are examples the writer chose to use. The interpretations made on how the framework might have helped are the writer's, and

have not been cross checked with other participants. The plausibility

of framework usefulness thus rests only on the events selected and the

interpretations provided by the writer.

In order to counter these limitations and biases, citations are

made to project documents to support the actual framework application. Judgements made by the writer as the application was conducted

do not contain supporting citations, unless an existing document was

found that corroborated the writer's opinions. It should also be
210

noted that specific names for all citations are not used, but are

replaced either with COTR, the name of the organization the person

was with, or the code names. Contractor A or C as appropriate.

Overall, it is felt that, even though there are potentially significant biases in the application, a reasonable case for framework usefulness has been made. This is because the application is based only

on events which are cited and not others which are either left out

of the application or unidentified in project records. Thus, the

plausibility of how the framework might have helped managers is based

only on the interpretation of events which have been adequately cited

in the text.

4.4 CONCLUSION

This chapter has reviewed the exploratory research process used

to develop a framework for monitoring system development and to apply

the framework retrospectively to a specific ETIP system project. The

first part of the chapter traced framework development from the early

ideas on management needs in ETIP system projects through to the resulting framework design. This description showed how the research problem

changed from the search for an ideal model of system development to the

formation of a framework for monitoring key areas of a system project.

The framework has been designed to be useful in project management as

a means for problem definition, strategy formulation, and the documentation and dissemination of progress. The framework is also useful as
a research tool since it provides a means for the study of important

system development problems and practices.

The second part of the chapter presented the design of a retrospective framework application to a specific ETIP system project. The appli

tion has been designed to illustrate the usefulness of the framework

in project management. This has been accomplished by developing procedures to first map the framework onto the project and then to show how

use of the framework might have facilitated the resolution of major

problems existing in the case.

The next two chapters present the results of these research efforts

Chapter 5 describes the 30 factors of the monitoring framework and the

five functions it may serve. A model for Implementing the functions

is also presented. Chapter 6 presents the retrospecti/e application

of the framework to a fourth ETIP project, the evaluation system

development.
212

CHAPTER 5

THE FACTORS AND FUNCTIONS OF A FRAMEWORK

FOR MONITORING SYSTEM DEVELOPMENTS

5.1 INTRODUCTION

This chapter describes a set of factors for monitoring a system

development and a set of functions monitoring can serve. The framework

is designed to help project managers collect information on key project

areas and use this information to identify problems, assess and revise

strategy, support decisionmaking, and help promote awareness of progress.

It also has a research function in that the monitoring process can be

used as a mechanism for conducting investigations of the areas being

monitored.

Thirty factors and five functions are identified for the monitoring

framework. The factors are divided into three main categories, which

reflect the overall types of factors which should be monitored. These

are design, process, and user commitment characteristics. The five functions which monitoring can serve include problem identification, strategy

development, research, documentation, and dissemination.

The following sections further describe the factors and functions

at a level which should facilitate access to the more detailed sections

of the chapter. A final section is also included to direct readers to

these more detailed discussions.
Readers interested in learning how the framework was developed

should consult Chapter 4.

5.1.1 The Monitoring Factors

As mentioned above, the factors in the framework are divided into

three main categories — design, process, and user commitment characteristics. These categories were chosen to reflect the general types of

factors which need to be monitored and the specific needs of the ETIP

situation for which the framework has been designed.

Design characteristics are factors which describe what a system

looks like and how it works. Monitoring these characteristics is

important for several reasons. First, the design must be sound, meaning that it actually provides the functions needed by users as well

as meet the specific performance requirements. Second, the design

must be attractive to users in order to gain their support and use.

This means that the design must help users solve important problems

in a manner which is matched to their style and other organizational

processes. Problems in these areas are common in system developments

and can jeopardize the ultimate success of the project. Developers

need to routinely monitor design characteristics and decide whether

revisions in the design or the project strategy are needed to ensure

that a satisfactory system is evolving.
214

Process characteristics are factors which describe the methods,

structures, procedures, and personnel used to conduct and control

the development. Monitoring these characteristics provides developers

with an indication of whether the process they are using to design

the system is producing the appropriate design and leading to full

implementation of the system at the end of the project. A widely

recognized factor in this category is the need to involve users in

the development as much as possible. Developers must decide for themselves what this means for a project and then ensure that the level

of involvement is achieved; ultimate user commitment to a system may

depend to a large extent on the earlier active involvement of users.

This is only one factor among many, however. Other parts of the development process such as the priority of tasks, the level of turnover

in key personnel, or the kind of reporting mechanisms used to document

progress may also have an important role. Factors like these can

be a problem at any stage of development and it is important that

they be routinely reviewed. Developers may find that they have implications for eventual user commitment to the system and/or system design.

The third framework category, user commitment, consists of factors

which characterize behaviors related to use and support for the system

in the prospective owner organization. Commitment is viewed here as

developing over time, making it essential to monitor it during the system

project and determine whether it is increasing or decreasing. The final

user decision to fully implement a system will be a reflection of the

commitment already established during development. For example, an early
sign of commitment would be the emergence of a system champion in the

user organization who strongly supports development and institutionalization of the system. Champions will help locate the key user problems

which need a new system as well as actively market the system to others.

In contrast, continuing problems in defining system functions in the

user organization may indicate an unusual degree of uncertainty or a

change in commitment to specific design concepts. Signals like these

need to be identified and considered by developers during a systems project and their implications determined for design or strategy.

Two figures are provided to summarize the factors of the framework.

First, Figure 5.1 lists the framework categories and all of the 30 factors. Second, Figure 5.2 provides a brief description of each factor.

These descriptions are also used in the more detailed discussions of

the factors.

5.1.2 The Monitoring Functions

The monitoring framework is designed to serve both administrative

and research purposes in a system development. Administrators can use

the information collected through monitoring to help identify problems,

revise strategy, make decisions, or to help make others aware of development progress. Researchers can use the monitoring process to collect

data for investigations of the development process.
216 1

COoMH

CO to M 4.5 Jffi CO CO c 03 M c 05 03 CO H 0 0 CO E 3)

CJ •1—

E-h c 40

•=£ CO 4-) CJ 0 •H 03 05 0 CD «a; •H E 55

•=3; 0 w <c to CI4 a E 40 c I H c 35 e 0 0 0 CJ 0 4-> E-I 03 CJ C •iH <X •H c 05 •rH 4J

E-i Cij 03 0 0 0 03 03 Z 03 3 Oh 0 CO N

Cd 0 CT 4-5 Ol^ E 2h 03 •H S CO •H CD c 35 03 3 00 c

E-i :j t—

CO (D CO 4-5 0 c 03 1—

a C 40 CO CO 03 bO

2: Q, 0 X 2 >, 03 -C !L, u CJ CjJ W CO 05 CJ 0 0 E-I E-i CJ CO • • • CO « • « >H LT> c— >H OO a^ 0 OS CO C^J CM CM CO CM CM m M

CO • • 3

05 0

Cd •3 CO T3 0 CO 4-5 CO CO 05 3 2 X 0

05 05 0 0 0 0

CO 0 0 Eh 0 w •fH Cd 4-5 •3 CJ El CO Eh 0 c 0 0 > X CO 3 Cm M tJi 03 CJ <=5 03 •H 4-5 0 Eh ffi 0 •3 El <=5 •r-( =3: Cx. 4J C 05 0 E 4-5 CO ClH 4-5 Cxh C •H •rH X Eh 0 3 M •H 33 0 rH 0 T3 Eh 0 > 0

05

M 3 J 4-5 0 •r4 •rH Oh 0 3 M =1: r3X

IjJ 05 03 0 W

C 05 40 33 0 0 s ClH 0 M

E-I 3) CO •r4 z 3 0 Eh 0 •rH 3 3 > 3

CJ Eh •H 3 z 0 E Z COW 0 0 0 z H 3 E bO 0

•=5 0 CO Qh 0 CO > 40 0 •r4 c •rH 4-5 4-5 0 Z M 0 3 Cm

05 3) CO f—

0 •H CJ) c 0 0 0 3 M Cd •3 0

•a; 05 05 3 E 0 a •3 0 0 Eh S 3 X 4-5

C

X Eh CO toW •rH 3 E H bOn 0 <—

a CJ Z 0 0 0 0 0 CO cd 03 Oh 3 0 0 3 0 0 •3 0 <c 0 0 3 0 3

Eh Eh OQ Eh CJ Cd 0 05

Q 2 05 05 05

X

X Eh Eh CO bd s 13) M M

CO CO • . «a: • . « 0 . . . H > • • . . w < 00 =r Cd LPl MJ l>- 05 OO CT5 0 Z Z T— CM m =r u Eh r— Eh T— t— T Oh ?

r— CM 3H

X CM CM CM CM 0

05 • • • •

Oh 0 0 E E 0 0 0

43 JH

X 0 0 X

>> 0 >1 bn 0 0 3 CO 0 0 3 X 3 X 0 3 JH 3 •H CJ 0 3 •H 3 3 0 bO X z H CO 43 •fH 0 •rH E •H 0

>=5 4-5 0 0 X

X 0 0 X S 0 X 43 X 0 4H 0 X 0 a

05 3 3 0 •H Eh

a 0 0 0 0 a

CO 0 0 0 E •fH 3 •H >, CJ •H a 3 •H T3 X X 0 CO X CO 0 X M « 4-5 0 0 X 3 0 43 •H X •H X 3

E-'

X 0 0 •iH CO \ 0 3 •H

X 0 0 0 X 0

CO

X E 3 43 4-)

X 0 0 0 2 0 0

)—•H 0 3 0 0 0 3 0 •H CO 0 C3XX 3

05

Q

H a.

a 0 Eh •fH 0 a 43 0 •fH =5 CO 0 0 W Z 0 43 3 0 43 •H 3 0 0 3 X bO •H Eh <5 0 3 3 < •H 0 rH 0 0 0 3 >1 X

CJ 0 >>

X 0 0 X rH 0 3 0 0 bO z rH •H

X 0

=5 Z 3 4-5 0 \ 0 *H

a 3 X 3 0 X

X rH 3

cc. 0 0 •H 43 3 >H

X 0 3 3 0 M •«H 0 3 0 rH

<£ 1—

CO O4 r—

4-5 3 0 X 0 0 3 0 0 Eh

X

X 0 > 0 X £h X 0 0 0 (X 43 a: a a 0 43 X X =5 0 0 0 0 > 0 < 0 0 3 0 3 3 Q 0 X 0 3 3 3 Eh rH 2: X z Cd « Eh X CD' X M 1—

z X X X 1—

M ca X CTo z X CJ X <=5 • • • 0 X <c 0 • • • Q • 0 T— CM

1—

0 Cz^ r— CM PO .=r inX CO CTi T— T— COX • • • Q

FIGURE

5.1

THE

MONITORING

FRAMEWORK
DESIGN CHARACTERISTICS

Operation and Performance Factors

General Description:

Factors

The functioning of the system design

in the user organization.

1. Response Time — The amount of time it takes a system to

respond to a user inquiry.

2. Quality — The accuracy, credibility, and utility of input

and output information in the system.

3. Cost of Operation — The amount of resources needed to operate

the system.

4. Input/Output Operations — The mechanics of user interaction

with the system.

5. Interconnection of Subsystems — The interrelationships of

system elements.

Boundary Factors

General Description: The borders of system and subsystem designs

Factors

6. Capabilities/Limitations/Expectations — The conceptual boundaries of a system prescribed by the various groups of people

involved in the development.

7. User Groups and Their Interrelationships — The roles and costs

and benefits of involvement with the system pertaining to the

different groups of the user organization created by the system

design.

8. Interfaces with Other Systems and Organizations — The relation

ships between the system and other systems and organizations.

Adaptation Factors

General Description: The match between the system design and the user organization.

Factors

9.

Flexible Specifications — The match between design specifications and the existing uncertainties over system objectives,

processes ownership, products, etc.

FIGURE 5.2 BRIEF DESCRIPTIONS OF THE MONITORING FACTORS
218

10. Matching the System to the User — The match between the system

design and the user organization structure (structural), the

abilities, methods, and personal styles of individual users

(technical), and the personal relationships of individuals

in the user organization (personnel).

11. Novelty of the Design — The change a new system represents

over the existing system.

12. Evaluation and Updating — The provision of evaluation and

updating functions in the system design.

FIGURE 5.2 (continued)
PROCESS CHARACTERISTICS

Task Structure Factors

General Description: The design of project tasks to develop

the system.

Factors

13* Task Size — The amount of resources (money, manpower, and

time) assigned and consumed in the performance of individual

system development tasks.

14.

Task Priorities — The relative emphasis across tasks at a given point in time and the ordering of tasks over time.

Team Personnel Factors

General Description: The availability and functioning of project

personnel.

Factors:

15. Skills — The availability of technical and interpersonal skills

in project teams as required by system development tasks.

16 . Turnover — The change of personnel involved with project teams.

17. Commitment — The team member support for and implementation

of the goals, strategies, and tactics of the system development.

Project Control Factors

General Description: The structures and processes used to

control project activities.

Factors

18 . Organization and Responsibilities — The structure of project

teams and responsibilities in the system development.

19. Decision Points and Milestones — The structure of specific

events created by project managers to recognize or review progress and decide future courses of action.

20. Reports and Reviews — The structure of written and oral communication mechanisms used to document and review development

progress.

Interaction with the User Environment Factors

General Description: The involvement and contact between the

project and members of the user organization.

FIGURE 5.2 (continued)
220

Factors

21. User Involvement — The participation of members of the user organization in the managerial and technical teams of the project.

22. Problem Identification — The overall amount of attention to

and the amount of contact with the user organization in the

definition of user problems.

23. Testing — The amount of concept and design testing conducted

in the user environment.

24. Transfer — The amount of attention devoted to the transfer

of the system to the user environment.

FIGURE 5.2 (continued)
USER COMMITMENT CHARACTERISTICS

System Use Factors

General Description: User actions related to application of the

system in organizational processes.

Factors:

25. Applications — The use of specific elements of the system

in specific units of work of the user organization.

26. Consequent Actions — The actions resulting from areas of

direct system use.

27. Extent of Use — The amount of use relative to the number of

potential applications.

System Support Factors

General Description: User actions related to supporting application of the system to organizational processes .

Factors

28. System Champions — The emergence of advocates for the system

in the user organization.

29. Resource Commitments — The type and extent of resources

allocated by the user organization to support the system.

30. Changes in the User Organization — The alteration of policies

and procedures in the user organization in order to support

system operation.

FIGURE 5.2 (continued)
222

These two main uses are combined into five functions for monitoring

which overlap and support each other. These are:

• Problem identification — The tracking and assessment of key

areas where development problems typically occur, both in the

short and long range.

• Strategy development — The identification and development

of explicit management actions to solve problems and revise

strategy

• Research — The design and implementation of real-time studies

of the development process.

• Documentation — The establishment of an organized and stable

recording process which can identify special or recurring problems, and support decisionmaking and research.

• Dissemination — The distribution of key development information to these inside and outside a development as a means of

facilitating coordinated actions and distributing knowledge

gained.

5 . 1.3 Organization of the Chapter

The following sections of Chapter 5 more fully describe the factors

and functions of the framework. Four main sections are used — the first

three describe the factors by category, and the fourth describes the

functions and implementation of the framework in a systems project.

Each category of factors is separately described. First, the category is generally introduced. This is followed by subsections in each

of the respective subcategories of factors (as shown in Figure 5.1).
Each factor is then described and cross references are made to other

factors which are relevant. The brief descriptions of the factors

1 provided in Figure 5.2 are included again in these discussions.

As mentioned in earlier chapters, several ETIP system development projects have been used as supporting illustrations in the sections describing each factor. The three cases are the Small Business

Administration (SBA) project to develop a data base system, the Public

Buildings Service (PBS) project to develop a planning system, and the

state and local procurement project to develop a procurement information system involving the National Association of State Purchasing

Officials (NASPO) and the National Institute of Governmental Purchasing (NIGP) . An attempt has been made to include sufficient background

material in these illustrations so that they can stand alone without

referencing to other sections; however, it may be beneficial for readers to review the case descriptions located in Chapter 2. These descriptions review the entire case and provide the contextual information in a more succinct manner. Reviewing Chapter 2 may make the

illustrations of Chapter 5 more quickly understandable and allow tlie

reader to skip material which is repeated. Also, readers will note

that the specific illustrations presented in Chapter 5 often go beyond

the details found in Chapter 2. This is intentional since Chapter 2

is intended to provide the general background which is too cumbersome for the illustrations needed in Chapter 5.

1 Readers will note that each category, subcategory, and factor

description is started on a new page. This has been done to

make it easier to quickly find any of the framework items.
224

Following the discussions of framework factors, the framework

functions are presented. First, each function is discussed separately.

A model for applying the framework is then presented to show how the

monitoring functions can be implemented in a project.
5.2 DESIGN CHARACTERISTICS (FACTORS 1-12)

5.2.1 Introduction

At any stage of development, a system can be measured on certain factors which describe what it looks like and how it functions

in the user organization. For the monitoring framework, these factors

are called design characteristics.

The following sections identify and describe 12 factors. These

factors were chosen for their importance to system development success

as identified in the literature or ETIP experiences. They also appeared to be relevant factors for the ETIP evaluation system project

discussed elsewhere in this dissertation (see Chapter 6). More specific discussions on the choosing of factors can be found in Chapter 4

(see section 4.2).

The 12 factors have been separated into three major categories:

operation and performance, boundary, and adaptation. Factors in the

operation and performance category are for monitoring how a design

is planned to function or how it is actually functioning in an organization. Factors in the boundary category are for monitoring the

various kinds of interfaces a design creates. These include the boundaries between the different user groups in the organization, boundaries between the organization and outside groups, and boundaries between the actual design and other "theoretical" designs. The final
226

category of factors is adaptation. These are factors for monitoring

the fit of a design to a user organization.

One section is devoted to each of the three major categories.

The sections first present general discussion on the factors in the

category. Then each factor is individually described and supported

with citations to systems literature and to the ETIP experiences discussed in Chapter 2.
5.2.2 Operation and Performance Factors

General Description

One category of design characteristics measures how the system

operates and performs tasks related to meeting information requests.

These characteristics have an impact on people using the system and

thus are ultimately linked to the success or failure of the system.

Monitoring them during development may decrease the likelihood that

inappropriate designs or poor performance stay uncorrected and act as

barriers to obtaining full support of the system.

Several key characteristics have been identified in system literature and ETIP cases. These are: response time, quality, cost of

operation, input/output operations, and interconnection of subsystems. Each of these is described below.
228

5. 2. 2.1 Factor 1 — Response Time

Description — The amount of time it takes

a system to respond to a user industry.

Discussion

The time it takes a system to respond to a user, usually in relation to a specific information request, is an obvious design characteristic which will affect use. It is commonly mentioned in the systems literature, often as a criterion for evaluating a system (Krasnican,

1971, pp. 5^-55). While it also often appears as "timeliness of output reports" (Lucas, 1975b, p. 3^), it also can refer to any reaction

a system would have to any user request. Long delayed responses might

be expected to gradually discourage a user from interacting with the

system. Quick turnaround might accelerate introduction of the system

and build user confidence.

ETIP Examples

Response time was definitely a factor in the ETIP cases described

in Chapter 2, particularly in the small business set-aside data base

system (see section 2.4). In that project, developers attempted to

build a system that would help users develop a case to set-aside a

government procurement to small businesses. Users normally had 15

days from the time they learned of a pending procurement to make their

case. As described earlier, a manual system was first created to
meet this need, but its time performance was unacceptable to users

and it did not gain their support. A computerized version constructed

later did solve the time problem, and this was partly responsible

for its acceptance.

Response time was also an important factor in the ETIP case

involving the PBS planning system (see section 2.2). While the computerized system did not meet the objectives of the project as originally planned, it did provide a new mechanism to calculate cost analyses, a subset of the problem. The computer program reduced calculation time from weeks to hours and also provided for much more flexibility in adjusting the assumptions used. Some users saw the value

of this aspect of the system and started to routinely make cost analysis requests. Thus, even though the system was not being used as

fully intended, it gained further support from users and remained

in place after the close of the project.
5. 2. 2. 2 Factor 2 — Quality

Description — The accuracy, credibility, and utility

of input and output information in the system.

Discussion

The quality of a system, particularly in its response to a user

request, is also one of the more obvious design characteristics for

systems. It is often used, like response time and cost, as a criterion for evaluating a system. It is considered one of the most important factors in a development since it can have significant influence

on user commitment to the system. A high level of quality (ensuring

a high commitment) is not always realistic however, since the benefits

obtained from it must be compared to the cost of achieving it.

For information systems, quality can mean many different things.

Lucas (1975b, p. 3^) defines information system quality as a measure of input and output including clarity of documents, input error,

usefulness, accuracy, and timeliness of output reports. Quality is

also associated with quantity to some extent. One of the major problems with an information system might be that too much information

is given to decisionmakers, or that they may want more information

than they actually need (Ackoff, 1967). The quantity of information

may need to be controlled and irrelevant information eliminated (Lucas,

1975b, p. 112). Indirectly then, quantity would be considered as
reflecting on quality also.

In this dissertation, quality is used as a measure of input and

output information characteristics, especially as to their accuracy,

credibility, and utility. These characteristics are important from

two perspectives. First, the system itself must be sound, meaning

that it provides the correct information. Second, the system must

produce what is needed by users and lead to their support and commitment to institutionalization. While these attributes can be related,

developers must ensure that the system has both since one does not

necessarily follow the other.

ETIP Examples

Quality as a general factor was important in each of the three

ETIP cases described in Chapter 2. In the small business case, the

project began to make significant progress when the computerized system output matched the results obtained independently and previously

by users. In the PBS case, the system was able to provide economic

analyses on space configurations of a quality at least equal with

the previous manual system. Since the computer system was overwhelmingly faster than the manual version, users quickly adopted the system

into their routines. Finally, in the state and local procurement

project, one of the original objectives was to upgrade the quality

of information available to procurement officials. These officials

often had few resources to do this on their own. However, system de-
232

velopers found it difficult to produce information at the level of

quality desired by users. Some officials found system output to be

less useful than what they could produce on their own. Other officials found it difficult to understand and/or accept the information.

Developers had committed themselves to a design which produced information of insufficient quality to attract users. Earlier attention

to this problem might have altered project strategy.
5. 2. 2. 3 Factor 3 — Cost of Operation

Description — The amount of resources needed to

operate the system.

Discussion

The cost of running a system is another obvious factor which

will affect user commitment. If a system is going to cost too much

to operate relative to its benefits or user resource constraints,

users will be less likely to support it. However, it may be difficult to know the costs in advance and avoid this problem, especially

if there is a little historical data on another systems or if there

are great technical uncertainties (Chestnut, 1967, p. 180).

Operating costs are of considerable importance in both hardware

and software systems. In weapons systems, a common major problem

is the tradeoff between the quality of the end product and the economy in operating costs (as well as in development and production costs)

(Peck and Scherer, 1962, pp. 461-580). Support of a complex, high

quality system must often be tempered by the realities of how much

it will cost to operate and maintain the system. Similar problems

exist for information systems. In Ruth's (1978) development failure

discussed in Chapter 3, cost was one of the original justifications

for attempting the development. It was thought by managers that the

system would reduce overall costs in Air Force logistics since it

would replace numerous personnel. The system was designed around
234

the innovative but previously untested idea of processing all logistics transactions quickly rather than by their priority. However,

this system proved to be too costly relative to management's objectives. This was evidenced by their decision to return to the old

inefficient system after some 15 years of development and $200 million

of expenditures.

ETIP Examples

A further example of cost and its affect on development process

and commitment can be found in the ETIP-SBA data base system project.

The computerized system developed in the latter half of the project

met some user needs and gained modest support. Unfortunately it was

too costly to operate; users could not acquire the resources to

support it. Some of them felt that the capabilities of the system

were too small relative to cost. (It is also interesting to note

that the project itself did not have the resources to continue supporting the system either.)
5. 2. 2. 4 Factor 4 — Input/Output Operations

Description — The mechanics of user interaction

with the system.

Discussion

Another factor under the operation and performance category is

the operation of system input and output functions. The literature

often includes this as a factor in the measurement of system quality,

but usually this is in relation to the quality of data put into a

system and the quality of information produced. The focus here will

not be directly on these measures, but on the actual operational characteristics of input and output.

The operation of input and output functions is important to look

at for several reasons. It must be mechanically easy for users to

interact with a system (Lucas, 1975b, p. 112). If it appears too

difficult for them to get a request in or out of a system, most likely

they will not use it or will find ways to go around it. The same

conditions apply to the personnel responsible for providing data to

a system. It may be difficult to collect the data or there may be

uncertainty as to how to put it into the system. These problems could

havq significant impacts on their support (Waller et al., 1976, pp.

13-14). For example, Malvey (1978, pp. 191-211) found that disgruntled

staff responsible for input data failed to provide it or allowed numer-
236

ous errors to remain in it. The important lesson, she concludes,

is that the design of input and output functions may conflict with

or enhance the way an organization works or solves problems (Malvey,

1978, p. iii). Developers should spend time considering this problem

(Waller et al., 1976, pp. 29-41).

ETIP Examples

The importance of input and output operations was demonstrated

in the ETIP case studies. In the SBA system project, initial designs

required users to spend much time collecting and structuring information about small businesses. The system would then store this data

for later rapid access when needed in set-aside cases. During pilot

testing, however, it was found that users didn't have the time to

perform the input procedures because they were usually busy just identifying potential set-aside contracts. There was also some question

as to the appropriateness of the input forms. They had the potential

of tipping off small businesses to an impending contract opportunity

before it was announced through normal procurement channels. Finally,

some users had difficulty with the input procedures themselves. A

reliable system to classify all firms and contracts could not easily

be developed. Each of these characteristics contributed to the lack

of user interest in the system.

A contrasting example was found with the input operation of the

PBS planning system. In this system, the economic analyses formally
done only by hand were now available in an on-line computer program.

Input procedures required typing in space configuration parameters

and other constraints. It was simple and easy to change numerous

assumptions in the program or to test slight alterations of the same

design. The simplicity and flexibility of the input process (and

of course the rest of the system) was attractive to users and helped

gain their support for the new system.
238

5. 2. 2. 5 Factor 5 — Interconnection of Subsystems

Description — The interrelationships of systems

elements

Discussion

One component of any systems definition is that system elements

are related to each other and function in a manner which meets a common objective. Besides the breakdown of elements then, a key characteristic of system development is the interrelationship of the various subsystems.

Elements which are related by an overall objective are not necessarily related directly by input and output flows however. Elements

may function independently of each other or be heavily dependent on

each other. Malvey (1978, pp. 251-253) found that one factor in the

failure of the MIS she studied was that the entire system had to be

run for each type of request made. It was impossible to run elements

(e.g., a sub-routine) separately and thus operations were costly.

An important characteristic to monitor then is whether and how

elements are connected to each other. Developers should consider

organizing subsystems such that interactions between them are minimized (Flagle et al., I960, pp. 96-97). Besides allowing for easier

removal and update of components, minimizing interactions may also
help minimize the impact of changes to the user organization or personnel relevant to these subsystems (Comptroller General, 1975, p. 17;

Fronk, 1978).
240 I

5 . 2.3 Boundary Factors

General Description

A system can be described by what is included and what is excluded. Similar descriptions can be used at the subsystem level.

The boundary of a system is an important concept to developers and

its placement can be very influential on development activities.

There are a number of ways to delineate the boundaries of an

information system. For example, Chestnut (1967, pp. 98-134) suggests three different approaches: chronologically, physically, or

functionally. Any of the various approaches can be helpful in formulating a design and their relative merits will not be discussed here.

However, there are two categories which can be used to generally classify the different approaches: physical boundaries and conceptual

boundaries. With information system, physical boundaries usually refer

to the actual input and output devices through which a system will

collect and disseminate information (e.g., terminals, file cabinets,

etc.). These design characteristics are separate from the issues

of interest in this section and are discussed under Operation and

Performance (section 5.2.2).

The main concern in this section is the conceptual boundaries

which define the system and the implications they have for the user

organization. Conceptual boundaries are the words, thoughts, plans.
objectives, expectations, models or roles represented in or guiding

design characteristics. Several different strategies have been selected to examine where boundaries are located. The first category

is capabilities, limitations, and expectations for the design. Developers need to monitor how well these match across and within different user groups. A second category is user groups inside the organ

zation and their interrelationships. A design may conflict with or

complement existing relationships and developers need to examine what

the implications of either might be. The final category is interfaces

with other systems and organizations. The system may eventually depend on or support outside groups, and thus the relationship between

the system and the groups may be a critical factor developers should

monitor to ensure project success.
5. 2. 3.1 Factor 6 — Capabilities/Limitations/Expectations

Description — The conceptual boundaries of a system

prescribed by the various groups of people involved

in the development.

Discussion

One way to identify the conceptual boundaries of a system is

to identify the capabilities, limitations, and expectations for it

in terms of design characteristics. How a system gets developed,

or acquires capabilities, is partly a function of the interplay between the expectations and limitations various groups have for design.

Capabilities in this sense are what a system is theoretically designed

to do, rather than what it actually does. Actual performance is covered in the earlier section on Operation and Performance (sections

5 . 2 . 2 ).

The important issue to examine is the gap that may exist between

these factors. Significant gaps may become a source of problems for

developers

• Limitations must be kept in mind such that the system's

anticipated capabilities are not oversold (Krasnican, 1971,

p. 55).

• Excessive expectations which are not met by actual performance may cause support to wither away or be unattainable

(Ein-Dor and Segev, 1978, p. 1072).

• User expectations which are lower than those claimed by

developers may cause the system to appear too grandiose

to potential users and gain little attention or credibility for it.
243

Conditions like these would be further exacerbated if designers

I and users see the gaps differently (Meador and Ness, 1974). DevelopI ers need to identify the key gaps that exist and think through the

!j implications they have for development strategy and system design.

I

I Many sources in the literature advocate the need to identify

I and monitor boundaries and gaps early. At the very least, initial

boundaries must be defined so that the information problem can be

I

I defined and components internal and external to the system can be

'' identified (Chestnut, 1967, p. 133; Henderson, Note 1; Zimmerman,

1977, pp. 184, 189).

|i It is also important to monitor these boundaries later since

unknown events may occur that affect them. They might shift due to

changes in user commitments to designs, economic pressures experienced by the organization from outside, personnel turnover, or because competing systems become available (Ackoff, I960; Thompson,

1978). Some development models, especially the newer ones used in

government (see section 3-3. 3. 2), provide for changes like these since

it is recognized that increasing knowledge about the problem and the

solution will likely induce changes. For example, the cornerstone

of the government model (A109) proposed for acquiring large scale

systems is the decision point at each stage where system capabilities

(planned or actual) are compared with original expectations. Significant gaps found in this comparison are supposed to indicate to developers a need to revise goals and strategy or cancel the project
244

altogether. Alternatively, developers may strive for a consensus

between competing positions (King and Cleland, 1975).

ETIP Examples

Gaps between capabilities, expectations, and limitations were

found in all three of the ETIP cases described earlier. For example,

one of the more significant gaps occurred in the SBA data base system project. The project had been developed around the concept of

improving the ability of SBA officers to match potential government

contracts with potential small business sources. This was to hopefully increase the number of small business set-asides being made

by the government. Developers constructed procedures to help analyze

potential contracts and small businesses in a similar manner, thus

improving the process of matching. After creating a system to do

this, however, developers found that the limitations in time and resources of field officials severely limited their ability to perform

these procedures and made it unlikely they would use the system.

What field officials needed and wanted was a simple system to help

them find sources to contact, i.e., identify names and addresses.

The boundaries of the two systems designs overlapped to some extent,

but user acceptance of the developer system seemed unlikely until

the design was molded to match user thinking. This gap significantly

altered the course of the project.

Another significant gap occurred in the state and local procurement system development. This was a case where exceedingly high expec-
tations in early stages were not met with actual capabilities in later

stages. Project developers originally thought that procurement problems at the state and local government level could be solved by promoting the exchange of information between those needing help and

those who could provide help. A simple exchange system was thus planned

and created. However, as described in Chapter 2, much more work was

needed to solve the user problems than just exchanging information.

Many of the procurement problems were based on institutional factors,

such as legislation or lack of money. Even in cases where there were

few institutional constraints, the system was unable to produce the

information needed because it had to be so closely tailored to specific sites. The internal technical expertise needed to do this had

not been anticipated by project planners and was not easily available

elsewhere. In the end, the gaps between limitations at the user sites,

initial high expectations, and the limited capabilities of the design

were insurmountable within the project and it was terminated. Earlier

attention to these gaps might have dramatically altered the goals

and strategy used by developers.
246

5. 2. 3. 2 Factor 7 — User Groups and Their Interrelationships

Description — The roles and costs and benefits of

involvement with the system pertaining to the different groups of the user organization created by the

system design.

Discussion

A system design usually creates a number of different groups

in the user organization. For example, there can be system owners,

input sources, output users, system managers, performance evaluators,

or system operators. These groups may cut across already established

organizational lines. Each group has a specific role in the system

and certain costs and benefits associated with it. Some groups may

even have multiple roles, such as when owners and users are the same

people. Another way to identify system boundaries then is to study

these groups and their interrelationships. Keen (Note 2) has considered

this issue by treating information as a commodity and then examining

territorial questions: owners, users, designers, evaluators, and

coalitions of these.

The issue of different user groups and roles is not widely discussed in the literature, particularly in relation to how developers handle the differences during a project. Many sources refer to

the related problem of high level support, e.g., that a high enough

level manager be in charge of the system so that coordination among
the various groups can be enforced (Ackoff, I960; Malvey, 1978,

p. 20). Others suggest that no one unit be allowed to capture or subordinate a system to their own interests (Ein-Dor and Segev, 1978,

p. 1074). Most sources agree that the design of a system must be different for different levels of an organization (Gorry and Morton, 1971;

Katch, 1978, p. 54).

Perhaps the closest example in the literature to the issue of

boundaries as seen in user groups is found in sources which describe

problems the various groups may have in supporting a system. For

example, Malvey (1978, p. iv) found that input sources had little

incentive to produce information for the system. In this case, the

costs and benefits of their participation may not have been in balance

enough to gain their support (see also problems with matching, section

5.2. 4.2). Similar cases are commonly found for users of the system.

Generally, the focus is on a single group and the conflicts and differences between groups are not discussed. Developers need to address

both the imbalances a design creates within a group (e.g., between

the cost and payoff of participating) and any resulting implications

for group interactions (e.g., cost/payoff differences between groups).

ETIP Examples

A good example of the problem is found in the state and local

procurement project. System developers assumed that owners and users

were the same people, namely, procurement officials in the executive
agencies of state and local governments. Later they found that this

was only particularly true. In order for the officials to acquire

the system, they had to obtain approval from other higher officials,

including distant legislators with control over appropriations. The

system was not designed with this group in mind and thus had no capability at the time for reaching them.

Another example was found in the PBS planning system project.

In this case, the system design was completed within a special studies unit of PBS outside of the line organizations intended to be the

ultimate users. The special studies unit maintained roles of owner,

manager, operator, and evaluator, while the real users had little

if any control of the resource. Even though the system attracted

users, the conflict between the units on system roles and organizational responsibilities remained as one factor preventing complete

user acceptance of the system.
5. 2. 3. 3 Factor 8 — Interfaces with Other Systems and Organizations

Description — The relationships between the system

and other systems and organizations.

Discussion

Another set of boundaries developers may need to examine are

those between the system and other systems and organizations (Ginzberg,

Note 3)* Outside systems or organizations may be sources of data

to the system or recipients of system outputs (King and Cleland, 1975,

pp. 288-289). Alternatively, they may have an indirect role through

control of system resources or competition with the system (Comptroller

General, 1978, pp. 9-10; Thompson, 1978, p. 329). The lack of an

interface with outside entities may isolate the system from mainstream

activities of an organization and reduce its importance (Nadler, Cammann,

and Mirvis , 1980, p. 56; Rice, Note 4). In any case, establishing,

monitoring, and refining relationships with these outside operations

may be important to system success.

ETIP Examples

An example of the interface problem was found in the PBS project. The system met with user resistance and developers turned to

outside groups with an interest in the system's methods. The system

had been designed bo analyze space acquisition alternatives for the

agency in a manner consistent with the desired approaches of acquisition policymakers outside of the agency. Thus when the system was
250

operational, developers began to market it to the outside policymakers

in an attempt to have it qualified as commensurate with their desired

acquisition policy. It was hoped that such an endorsement would further facilitate implementation of the system in the agency. In this

case, an effective interface had to be established with outside groups.

Early recognition of this situation allowed developers to orient the

system design with this fact in mind.
5.2.4 Adaptation Factors

General Description

The third major category of design characteristics to be monitored relates to how well a system is adapted to the organization.

A key problem in system developments is that an organization may be

force fit to a system, rather than a system being matched to the users.

This can create serious side effects in the organization and reduce

the likelihood of success. On the other hand, large scale developments probably will identify weak spots in an organization which should

be improved. In this case, changing the organization becomes a valid

strategy in developing the system. Developers need to monitor how

well the system and organization are fitting together and attempt

to identify symptoms of a poor match early before problems arise.

Developers also need to integrate the procedures to examine the match

into the final design so that they can be used to continually improve

the system during its operational phase.

Four major subcategories have been selected to further examine

acaptation. First, the area of flexibility in design is reviewed.

Flexibility is a lesson commonly mentioned in systems literature and

the main concern is usually to ensure that a design is not finalized

too early. Next, the need to match a design to the organization is

discussed. The main goal of keeping a design flexible is to increase

the likelihood of successfully matching the system and the organize-
tion. The degree of matching may be used as one test for determining

whether a design is too advanced for that stage of the project. A

third area is the amount of change a design represents relative to

an organization. Designs which are significantly different from the

existing system may be too radical for gaining user acceptance. Incre

mental changes might be needed in order to gradually adapt users to

new procedures. Finally, the importance of institutionalizing updating functions in the design is discussed. The methods used by develop

ers to review and revise a design should become part of the finished

system. These procedures also need to be tied to other user evaluations of the system and to updating initiatives evolving from changes

in the organization or its environment. Evaluation and updating functions should be thoroughly integrated in a design prior to its final

acceptance by users.
5. 2. 4.1 Factor 9 — Flexible Specifications

Description — The match between design specifictions

and the existing uncertainties over system objectives,

processes, ownership, products, etc.

Discussion

A lesson for developers commonly mentioned in the literature

is that a system design should be kept flexible in order to accommodate the uncertainties likely to arise throughout a project. Developers are advised to keep system specifications open or general, especially in early stages where system objectives are evolving. Specifications in this case are "living documents" (Chestnut, 1967, p. 64)

which are not "cast in concrete" too early for the situation (Lucas,

1978b, p. 51). Developers are also advised to provide flexibility

in system specifications so that the system can meet different user

styles. This problem is considered in the following section on matching the system design to the organization.

The flexibility of system specifications is a difficult concept

to operationalize for the purposes of monitoring. Specifications

are basically descriptions of the system from different perspectives:

technical, performance, organizational, processes, products, etc.

(Haynes and Wheelwright, 1979). Flexibility is a quality specifications have relative to the degree of uncertainty in these areas.

Uncertainties might exist from a lack of knowledge about an area.

For example, unless a system is to handle routine, well-known opera-
254

tions, it is very likely that the kind of output needed is unknown

even to the user. Other uncertainties could evolve from changes which

occur during the project, such as in personnel, organization, or budgets. Whether a system design is appropriately flexible relative to

these uncertainties is a difficult judgement and it is hard to quantify in advance. It should also be noted that in cases where there

are few uncertainties or changes, flexibility may not even really

be an issue.

This difficulty is reflected in the literature in that most

sources resort to recommendations on process rather than criteria for

measuring the flexibility of a design. For example, some sources

advise developers to initially focus on performance specifications,

or missions, rather than specific designs. This strategy is supposed

to allow the various options and uncertainties to be freely explored

before decisions are made detailing designs. The large scale system

acquisition model developed by the Office of Federal Procurement Policy (OFPP) uses this idea (see section 3*3.3.2). Early stages of the

model focus on missions of the agency and gaps in operations. Competition among designs is also encouraged as far into a project as possible.

The ideas behind these models are useful for enabling developers

to provide in advance the ’’climate" needed for design flexibility.

However, developers still need some way to gauge the design flexibility

required as a project unfolds. One approach is to study the sensiti-
vity of the design to the various changes expected during a development (Mumford et al., 1978, pp. 245-246). Developers should identify

what changes could occur in personnel, objectives, or technical uncertainties, etc., and think through what implications they might have

for the system design. Areas subject to great uncertainty or change

will be ones where a design should remain changeable.

ETIP Examples

The importance of monitoring a quality like flexibility can be

found in the ETIP development cases presented earlier. Generally

it was difficult to retrospectively identify the degree to which speci

fications were flexible at a point in time in the cases. However,

events occurred which indicate that a flexibility problem probably

arose. For example, in the SBA data base project, developers committed themselves early to a manually operated design which later proved

to be unpopular with users. This in turn stalled development since

it had been expected that the design would be field tested extensively

with users in order to refine its operation. Different strategies

were attempted to gain user acceptance and field sites, but few test

cases were obtained. It wasn’t until a computerized design, somewhat

different from the manual version, was completed that users became

more interested. In retrospect, it would appear that developers final

ized the design too early. The problems uncovered during the field

stage indicate that there were many problems unanticipated by developers which had significant implications for project success. These
256

uncertainties should have been reflected in a more flexible design

which would have allowed developers to understand the situation before

committing too many resources to one approach.

A very similar situation occurred in the state and local procurement project. Developers committed themselves to a set of objectives

and system specifications early in the project before adequate knowledge of the problem was obtained. As described earlier in the case

study, developers found that the design would not solve the user problems they thought it would and that gaining wide user support for

it would be difficult. Developers attempted to compensate for the

design deficiencies by allowing more variety in outputs and by attempting to acquire the more diverse inputs needed. Unfortunately, the

project had not been planned with these contingencies in mind, and

eventually it had to be terminated. Again it would seem that, relative to the uncertainty that must have existed at the time, the design

was frozen too early.
5. 2. 4. 2 Factor 10 — Matching the System to the User

Description — The match between the system design

and the user organization structure (structural),

the abilities, methods, and personal styles of individual

users (technical), and the personal relationships of

individuals in the user organization (personnel).

Discussion

The previous section outlined the need to keep a design flexible

in order to accommodate the uncertainties likely to exist in a development. The major goal of monitoring flexibility is to increase the

chances that the eventual design will match the organization and be

accepted by users. Thus, as designs are created and detailed, developers need to also test the match between the design and the user and

to determine whether satisfactory procedures are being developed.

Developers need to avoid the problematic strategy of forcing users

to adapt to a design (Malvey, 1978, p. iii; Ruth, 1978).

Matching can be analyzed at many different levels, however, and

a comprehensive procedure for monitoring them all is probably too

complex and inefficient. For these reasons, three categories which

seem to capture the major overall concerns have been selected for

discussion here. These are structural matches, technical matches,

and personnel matches (adapted from Thompson and Rath, Note 5). The

categories overlap to some extent and probably are part of any matching problem that arises in a development. They are described
258

separately below.

Structural matching concerns the match between the structure

of a system and the structure of the user organization. The way an

organization works has important implications for a system design.

Each user unit has its own goals, priorities, and ways to conduct

its work. Similarly, the organization as a whole has established

patterns of communications, relationships, and responsibilities.

It will be hard to develop and institutionalize an information system

which intrudes upon or ignores these existing conditions (Ein-Dor

and Segev, 1978 ; Keen and Morton, 1978 , pp. 69 , 107 - 108 ; Malvey, 1978 ,

pp. 264-265). On the other hand, a development may highlight deficiencies or previously unknown factors in how an organization works (Argyris,

1971 ; Krasnican, 1971 ). In this case, it may be beneficial for developers and users to consider changing the organization’s structure

as one part of developing a new system. McGrath ( 1970 ) for example

reports on a development in which corporate headquarters learned it

would have to recentralize itself in order to regain control of the

operations it wished to include in a new system. Some user groups

might of course withhold their support for a system if a change like

this is proposed. In any case, the match between the system and the

user is one design factor which developers should examine.

The second category, technical matching, concerns the degree

to which a system design matches the abilities, methods, and personal

styles of individual users. Individuals perceive and use information
differently (Churchman and Schainblatt, 1965; Lucas, 1975b; Mason

and Mitroff, 1973)* Some individuals are more data-oriented while

others are more intuitive. For example, top level managers probably

need a design which is directed toward better processing features,

like filtering and condensing, rather than one which merely overloads

them with more detailed information (Ackoff, 1967; Meador and Ness,

1974). In contrast, people at operational levels need a design which

is detailed and rigid, since more data is handled at this level and

the same procedure will be routinely repeated. It is generally recognized in the literature that incompatibility between user habits,

strategies, or abilities and the implicit ’’style” of the system will

result in decreased system use (Keen and Morton, 1978, p. 73; Krasnican,

1971, p. 56; Zand and Sorenson, 1975, p. 545).

The third category of matching is that of personnel matches.

This is the degree to which the people involved in a system are compatible with each other. Beyond matching a design to organization

structure and to individuals’ abilities, developers should examine

whether system personnel will be able to work together as required

and accomplish system objectives. System developments can fail because the relationships among the people are ignored, while too much

attention is placed on technical problems (Keen and Morton, 1978,

pp. 70-73).

This theme has recently been gaining support as behavioral scientists increase their attention on the organizational change problems
260

associated with system development. New ideas on implementation,

resistance to change, and organizational politics have been evolving

for system projects (Keen and Morton, 1978). Much of this concern

focuses on the user-developer relationship, especially since it is

commonly assumed that both people belong to the same organization.

Churchman and Schainblatt (1965) advise that users and developers

need to develop a mutual understanding; Lucas (1978b) proposes a model

for development based on this concept. This perspective needs to

be extended to other people in the user organization as well.

ETIP Examples

The PBS planning system project demonstrated the problem of a

mismatch between system and user structures. In this project, a system was built to help analyze the different options available for

meeting Federal office space needs. The system was conceived, developed, and operated within a special studies staff unit of the agency.

This group was attached to the upper levels of agency management and

normally did not have a role in the analyses. The users of the system

were the agency line divisions who did not have the responsibility

for the analyses. To some extent, these users were involved in the

development and did eventually start to use portions of the system.

However, their lack of control over the system highlighted a role

conflict that prevented full implementation.
The importance of a technical match between design and user is

easily demonstrated in the ETIP-SBA data system project. Here, developers created a system to help users match potential government

contracts with small business who could perform the work. A manual

processing design was first used which required a lot of information

about the contracts and the firms in order to perform the analyses.

Unfortunately, the design was poorly matched to the circumstances

users faced. First, users didn't have the time to do the matching

because of the short schedule agencies allowed them to have. Second,

they often could not get a complete, detailed view of the potential

contract because the agency would not release them in full before

the official announcement. Third, in order to make a match with a

firm, a user had to outline the potential contract to the firm and

gather information on their experiences with similar work. This was

a problem because it exposed the contract before it was officially

distributed by agencies, possibly violating their procurement procedures. Finally, users really wanted a system to help them find firms

They considered the problem of matching firms to contracts as part

of their expertise and skill and not a process which could be routinized on a machine. All of these technical problems contributed to

the unpopularity of the system and its later abandonment.

Demonstrating the importance of personnel matching in the ETIP

case studies is difficult since the evidence for problems is indirect

However, the state and local procurement project offers some experiences from which personnel matching problems can be inferred. In
this project, success depended heavily on the personal rapport and

influence developers could establish with potential users. The developers, NASPO and NIGP, were two nationwide professional organizations representing state and local procurement officials. While they

had no power over their membership, their professional relationship

with officials was expected to be sufficient to attract user attention

to the system and gain support. ETIP contracted with these two organizations and planned to have them assume full control of the system

at the end of development. Thus NIGP and NASPO, and their relationships to users, were to become integral parts of the final design.

Both NASPO and NIGP also expected to increase their stature and influence with officials by implementing the system.

Later progress did not fulfill these expectations. It was found

that users were influenced by a host of factors and that the developers did not always have the leverage to effectively cause change and

gain system acceptance. Neither NASPO or NIGP were able to motivate

users to conduct many of the experiments ETIP had planned. System

products became hard to develop and hard to sell to users. Technical

problems with these products further decreased NASPO and NIGP influence.

It was also learned that the financial support planners had expected to obtain from users in the long run was dependent on decisions

by executives or legislators outside of the immediate user agencies.

NASPO and NIGP had little established contact with these groups.
Thus not only was their influence weaker than expected with procurement officials, but they also had little prior experience working

with other key system supporters. Besides indicating a weak match

for personnel, this also unfolded into a broader structural problem

as defined earlier.
264

5. 2. 4. 3 Factor 11 — Novelty of the Design

Description — The change a new system represents over

the existing system.

Discussion

Another key indicator for monitoring the adaptation of a system

to an organization is the degree of change a new system represents

relative to the existing system (Thompson, 1978b). Incremental changes

would mean simple, small improvements from current procedures, while

more significant changes would mean the creation of previously nonexistent procedures. Generally, one would expect novel changes to

be more difficult to make. The resistance to change would probably

be higher in users and the extent of implementation problems greater.

In contrast, incremental changes would be expected to cause fewer

problems for users and generally be easier to implement.

Since there are many types of change involved in system development, developers need to review a number of different aspects in order

to gauge the degree of change involved. Significant changes in any

of the following design factors might have important implications

for development strategy (Thompson, 1978b, p. 329):

• System functions.

Functions previously unavailable may represent a significant

change to users. These functions may have been impossible
to develop before, or maybe they were not considered. There

may have been no preexisting need.

• Performance.

Dramatic changes in performance over an older system may

represent a source of significant change to users. While

the actual performance improvement could represent a small

design alteration, the implications of it may be much broader for all associated activities.

• Operation.

New ways of doing things may be a significant change for

users familiar with the older system.

• Other systems.

The relationship to other systems may change with the implementation of a new system.

• Support.

The organizational support for a new system might be very

different than that given to the old one. A new system

may create previously nonexistent barriers to its operation.

The importance of changes in these areas is well recognized in

systems development literature, especially that evolving from behavioral and management scientists (Keen and Morton, 1978, pp. 61-98).

Concern has been increasing for the impacts new systems can have as

a result of the changes they make in an organization. User resistance

to change and the broad area of implementation are two examples of

literature sectors examples where discussion on change can be found.

Several development strategies have been created specifically

around the problems associated with the degree of change involved

in a new design. Most of these strategies emphasize or imply slow,

incremental change from the old to the new system. New designs should
266

start out as only modest alterations of procedures or structures users

currently have. More complex functions can be added later (Alter

and Ginzberg, 1978; Lucas, 197^b). The implication of this strategy

is that fewer organization changes, and thus problems, will occur.

Further elaborations of these ideas can be found in other similar

models: the incremental approach, the bottom-up approach (Davis, 1974,

pp. 409-410), or the evolutionary approach (Lucas, 1978b; Mohan and

Bean, 1979, p. 147).

Other authors disagree with this strategy. Churchman (1954,

pp. 162-163) points out that the simple, incremental strategy assumes

that the "simple'* design can be identified and that progress from

simple to complex designs is possible. Mohan and Bean (1979, p. 147)

suggest that the evolutionary approach may be most effective in early

stages and that later on it won't make much difference in success.

They believe that a revolutionary approach, e.g., one involving radical design changes, may be appropriate in cases where there is strong

commitment for it from the user. Radical change might provide benefits more quickly than would be possible in a slower strategy, especially where it is obvious that a weak existing system cannot be improved

significantly. This was shown to some extent in the development reported by Ruth (1978) (see section 3»3.1.3).

ETIP Examples

All of the ETIP system development cases involved one type of

design improvement or another over the existing systems. Most of

these changes would probably be considered somewhere between incre-
267

mental and modest, since new designs did not significantly alter any

established functions. However, this judgement might vary depending

on who's perspective is taken.

For example, project planners probably thought that the proposed

SBA data base system was a modest change in operations for SBA field

officers. SBA wanted to increase the number of government contracts

set-aside for small businesses. In order to do this, they had to

improve the process by which field officers justified set-asides to

contracting agencies. This involved helping the officials more easily

match contracts to firms and demonstrate to agency decisionmakers

on objective criteria that qualified small firms existed. Essentially, users (the field officers) did this work already, except that

it was at a more brief, informal judgement level. Developers expected

design stages to be short and that most of the project would be field

testing.

Some users did not consider the new design to be quite so modest.

First, they felt the problem of matching firms and contracts was complex and required their skill and judgement. It was a process that

couldn't easily be formulated into a set of standard procedures.

Second, the forms and procedures associated with the system represented a lot of additional work to users which they didn't have time to

perform. As a result, very basic resistance to the design evolved

and developers were prevented from gaining further acceptance of the

system in many of the sites they were working with.
268

At another site however, this resistance was minimized when developers were able to give the users something they really needed. In

this case, developers decided to supply a data base of firms, thus

alleviating the significant problem of just locating sources. In

addition, the matching procedures were computerized, also eliminating

the time problem. These improvements apparently overshadowed the

concern about routinizing field officer skills since the system was

then used in real cases.

This case seems to demonstrate that design improvements over

an older system should be monitored for the degree of change they

imply. There will be different perspectives on the degree, however,

and each should be noted. The case also demonstrates that different

facets of change can affect each other. A change in one area, such

as a function (e.g., matching firms and contracts in the SBA case),

may be compensated to a degree by change in another area (e.g., including a new data base in the SBA case).
5. 2. 4. 4 Factor 12 — Evaluation and Updating

Description — The provision of evaluation

and updating functions in the system design.

Discussion

A final design characteristic for monitoring the adaptation of

a system to an organization is the preparation made in the design

for routine evaluation and updating. Users will most likely plan

for some kind of evaluation before they fully commit themselves to

a system. Periodic reviews of this nature may even continue after

the system is made operational. In addition, changes in the organization and its environment are likely to occur throughout the life

of a system. A design may become quickly outdated unless procedures

for revisions are also established. It will be advantageous if provi'

sions for these review and updating functions are considered early

in designs and made an integral part of the system (Jue et al., Note

6, p. 1; Waller et al., 1976, pp. 63-64; Zimmerman, 1977, p. 190).

Developers may want to monitor several different aspects of the

revision problem in order to assure that designs are being appropriately prepared. Some of these aspects are the following:

f Evaluation criteria.

Assuming that some kind of evaluation is needed, developers

should identify the decisionmakers involved and their criteria. These decisionmakers Diight include groups well out-
side of the immediate user units. These criteria may be

uncertain in early stages and may need development. Developers should think through how these criteria can be applied

and built into a design.

• Decision points.

Several models for development (see Chapter 3 and Project

Control, section 5 . 3 -^) suggest the need for explicit decision points at the end of each stage for approving the

continuation of the project. If this approach is used

in a project, developers should examine early what information might be needed about the system. Implications from

analyses like these may possibly provide the basic strategy

for later system testing (see Testing, section 5 . 3 * 5 . 3 ).

• Other organization review processes.

If the system is to become an integral part of an organization, its evaluation and updating functions may need to

be tied to already existing organizational review procedures

(Office of Federal Procurement Policy, 1976). This might

include reviews of the organization conducted from the

outside as well. Developers may have to adjust designs

to accommodate changes from these sources.

• Freezing a design.

At some point it may be advantageous to freeze a design

in order to bring the full system into operation without

continuing distractions for revisions (Flagle et al., I960,

p. 114). Developers need to examine where updates should

be made and which ones can be delayed without risking problems. In cases where updates are put off, provisions should

be made for users to initiate them when they assume complete

responsibility for maintenance of the system.

• Mechanisms.

Finally, developers need to examine how evaluation and updating function overall. They should study whether problems

are identified, whether redesigning occurs in a timely

manner, and whether changes actually are, or can be, made.

This mechanism must be operational like any other system

component when the system is finally institutionalized.

ETIP Examples

The kind of preparation suggested for this evaluation and updating indicator was not a significant part of any of the ETIP system de-
velopment cases. While evaluation and updates of the designs were made

during the projects, provisions were not made however to routinize

these functions within the designs. Only in the PBS planning system

were updating procedures considered in the overall design. Even so,

this meant only that it was possible to revise the computer programs

and that the agency systems department was given control to do so.

There wasn't an overall process tying system operation, evaluation,

or performance together in a manner which would promote needed and

timely revisions. The evidence seems to indicate that revisions were

expected to be performed on a more informal basis.
5.3 PROCESS CHARACTERISTICS (FACTORS 13-2^1)

5.3.1 Introduction

A system development can be described by the methods, procedures,

personnel, resources, and structure developers use to operate the

project. These factors contain a process perspective in that they

are the means toward an end: the eventual design and institutionaliza

tion of the system. Factors which denote project activities like

these are defined as process characteristics in the monitoring framework.

The following sections identify and describe 12 factors. These

have been separated into four categories: task structure, team person

nel, project control, and interaction with the user environment.

Task structure factors are for indicating how tasks are defined and

prioritized. Team personnel factors refer to characteristics of the

project teams developers use to conduct the tasks. The three main

factors in this category are the skills needed on teams, the turnover

of personnel, and the related factor of commitment to the project.

Project control factors indicate how tasks are being managed. These

include the organization of the central management team, delegation

of responsibilities, and mechanisms developers use to monitor and

review project activities. The final category of factors, interaction

with the user environment, characterizes the contact between users
and developers. This especially includes the key factor of user involvement in the project.

A special note should be made here concerning the different groups

which may be involved in a development. A development may be the

responsibility of several different institutions which are normally

independent of each other. For example, developers may need to enhance the technical skills on project teams because of the lack of

in-house staff. Thus, contract support may be obtained for the duration of a project or for special selected tasks. In another case,

developers may be supported financially by a third party, such as

an ETIP, even though this group is not a part of their organization.

The development process under multiple group arrangements like

these can be complicated. Authorities and responsibilities must

be delegated among the various parties and relationships must be established. The different roles and capabilities must be closely coordinated so that project resources are allocated efficiently. Different

perspectives on strategy may arise, as may other conflicts, and these

will have to be identified and resolved in order to maintain momentum

and commonality.

Complicating arrangements like these are not always considered

part of the development process. For example, contracting as an issue

has rarely been considered in the behavioral and management science

literatures dedicated to system developments. Models from these sec-
274

tors usually group everyone involved into either user or designer

categories (Kolb and Frohman, 1970; Lucas, 1978b) and ignore situations where independent parties may be involved. On the other hand,

contracting has long been a concern in military weapon system developments (Gordon, 1980; Hill, 1970; Livingston, 1959; Peck and Scherer,

1962 ).

Some of the following sections for each factor include discussion

on how multiple group situations can affect a project, particularly

in contracting cases. These discussions illustrate the additional

characteristics developers should monitor when more than one group

is involved. Many of the ideas in these sections have been adapted

from the techniques used in military weapon system acquisition procedures.
275

5.3 «2 Task Structure Factors

General Description

A system development can be divided into a series of specific tasks

or group of tasks. Tasks are usually grouped into categories or phases,

such as problem identification, design, or evaluation. These demarcations are fairly arbitrary, however, and disguise the true cyclic process

and sequencing of development tasks.

Of more interest are the actual tasks which occur within and across

phases like these. Identification and coordination of these tasks represent a significant problem to developers. For example, tasks related

to different subsystems may be similar and need coordination. In other

parts of the design, some subsystems may need to be fully developed before others. Task structure, then, is an area developers need to monitor

in order to assure that the various development activities are properly

sequenced and coordinated.

Two different characteristics of task structure are discussed in

this section. One is the size of tasks in terms of manpower, money,

and time. Developers should monitor tasks so that these resources are

consumed as planned. Large changes in consumption may have significant

implications for other development tasks underway or planned.
276

A second characteristic to monitor is the priority of tasks. Development tasks will have some order in which they should be performed.

This order might be established by factors such as resource availability,

time needed for accomplishment, or the priority of user problems. Since

any of these conditions can change, it is useful if developers monitor

them so that changes in priority can be quickly identified and

controlled
5. 3. 2.1 Factor 13 — Task Size

Description — The amount of resources (money, manpower,

and time) assigned and consumed in the performance of

individual system development tasks.

Discussion

The size of project tasks, or groups of tasks, is one characteristic

developers should monitor throughout a project. The size of a task includes the amount of money, manpower, and time consumed in performance

of the task. These characteristics together represent the "cost” of

developing system components, a related but distinct factor from the

cost of operating a system (see section 5. 2. 2.3).

One of the main reasons to monitor size characteristics is to control costs. Costs can easily expand beyond initial expectations if there

are major technical uncertainties in the design or numerous changes of

course in the project. Significant amounts of project resources may

become tied into solving these problems without considering the implications for overall project goals. Developers may thus reach a point where

they become locked into a design because most of the resources have been

consumed in trying to make it work. The lack of success and the need

for additional resources in order to finish the design may make further

work less attractive. Early and progressive monitoring of resource expenditures may help avoid becoming overcoramitted to a narrow portion of

the project or running out of resources at key points (Jue et al.. Note

6, p. 1; Flagle et al., I960, p. 110; Thompson, 1978a, p. 129).
278

While there is no one right size for project tasks, many literature

sources recommend dividing projects up into small, manageable steps which

are easier to control (Chervany, 1978, p. 157; Krasnican, 1971, p. 55).

There are some good reasons behind this strategy:

• Overall , system developments are complex undertakings

which involve numerous, intertwined problems. Breaking

off and working on small pieces of the problem can make the

project easier to handle. As more is learned about the user

environment, more complex and expansive tasks can be undertaken.

• In the early stages of a development, smaller investments

of resources by users and developers can reduce the risks

involved in choosing one direction over another. The chances

of a large scale failure may then be reduced (Davis, 1974,

p. 408).

• Developments are likely to contain a number of changes which

can upset the flow of a project. Using smaller, discrete

tasks and gradually moving towards more complex undertakings

can help protect against major disruptions (Fronk, 1978).

• Overall success may be more easily attained when smaller lines

of work are used. A strong track record of modest successes

may increase the likelihood of overall success (Edelstein and

Melnyk, 1977; Ein-Dor and Segev, 1978, p. 1073).

On the other hand, smaller, more numerous tasks imply more project

control and monitoring than might otherwise occur. More effort must

be expended to coordinate activities and to decide how to break projects

apart. This could become most inefficient if carried to an extreme.

Developers have to balance the risks in committing themselves to certain

task sizes with the benefits they can achieve in work flow, control.

and likelihood of eventual success.
ETIP Examples

Demonstrating task size problems in the three ETIP cases is somewhat

difficult in a retrospective analysis. All of the cases experienced

cost overrun problems. For example, in the SBA data base system development, the project ran out of resources before enough successes could

be achieved to gain the needed user support. Monitoring task size, among

other factors, might have helped avoid these problems, but it is hard

to show.

The state and local procurement project does, however, provide some

indication that developers were concerned about the task size issue.

In this case, they discovered that resources were being spread too thinly

across most of the major project tasks. Progress was very limited overall. In some areas, tasks were being ignored and resources needed for

their accomplishment were being jeopardized by commitments in other areas.

Developers eventually elected to reduce the number of tasks in the

project to a much lower level. It was hoped that more limited successes

would be possible before project resources were expended. Unfortunately

this did not occur and the project was subsequently terminated. In retrospect, it appears that developers may have become overcommitted to a

strategy which diffused their resources too broadly.
280

5.3 *2.2 Factor 14 — Task Priorities

Description — The relative emphasis across tasks at

a given point in time and the ordering of tasks

over time.

Discussion

A second characteristic of task structure which developers should

monitor is the priority of tasks. This would include monitoring the

emphasis and concentration needed across tasks at a point in time

as well as the ordering of tasks over time. Besides influencing the

flow of work, priorities can indirectly affect the level of support

and interest that a project gains, both from the user and the developer. Gaps between the user and developer priorities may reduce the

likelihood of successful development.

There are a number of factors which can affect what the priorities should be over the life of a project. One factor is resource

availability. Developers should examine when project resources, like

manpower and money, will be available. Project resources may be under

various controls which affect the time at which they are available.

This availability should be compared against the requirements of development tasks as they are planned and initiated. Developers may

also choose to construct resource controls of their own while performing tasks. For example, it might be advantageous to be able to make

some early accomplishments which are at a low cost to the project
and the user (Kraemer and King, 1978, p. 30).

A second factor affecting task priority is the time needed to

accomplish a task. There may be significant technical uncertainties

concerning design requirements. Designs which are significantly different from current user procedures may also create a number of other

organizational problems (see section 5. 2. 4.3). Developers should examine how long it may take to accomplish tasks and gauge their initiation

accordingly. Changes in time requirements during a project should

also be monitored so that implications for other tasks can be identified.

A third factor is the priority of user information problems.

Problems of concern to the user organization must be identified and

prioritized so that they are solved within a reasonable time horizon

(Mohan and Bean, 1979, p. 147). These problems would of course have

to be compared against overall project strategy. Major objectives

should not become jeopardized by distractions on numerous quick-turnaround user requests. On the other hand, developers should avoid

becoming mired in comprehensive information flow analyses which consume much time and produce few measurable changes. Some mixture of

attention to high level concerns (the "top-down” approach) and to

basic operational problems (the "bottom-up" or evolutionary approach)

is probably needed (Davis, 1974, pp. 405-409).

A fourth factor affecting priorities is the interrelationship

of subsystem components. Some parts of a system may need to be de-
282

veloped before others, while other parts can be developed independently or in parallel. As a consequence, development tasks may need some

level of sequencing in order to obtain a finished design. Early attention to boundaries at the system and subsystem level will facilitate

this ordering (Hill, 1970, p. 131).

A final factor affecting the priority of tasks is the need for

accomplishments. In order to maintain momentum and build credibility

for a design, it is useful to assure that measurable progress is made.

For example, it may be important to have some early outputs to which

users can react (Edelstein and Melnyk, 1977; Krasnican, 1971, p. 55;

Moore and Byrd, 1977). These might include dummy mock-ups of reports

or conceptual designs of the system. Items like these can help reduce

the early abstract level of a development and refine the user information problems. They may also help build top management support and

interest (Mohan and Bean, 1979, p. 147). Developers should monitor

where and when accomplishments are needed so that adequate preparation

can be made for them.

ETIP Examples

Several different examples of priority problems are available

in the ETIP system development projects discussed in Chapter 2. In

the SBA case, developers initially focused on the wrong user problem.

First they attempted to establish a system to help match federal contracts with potential small business sources. Unfortunately, the
priority problem for users was simply to find sources they could contact. Matching sources with upcoming contracts was a function they

had little time to do and considered an important skill. Developers

eventually had to accommodate their need before any significant progress was made in other areas.

In the state and local procurement case, a split over priorities

among members of the development team inhibited progress in later

stages of the project. Both sides had originally set the priority

in development as one of making early accomplishments with users.

This meant identifying several user information problems, developing

packages to help solve them, and actively marketing the packages to

users. Unfortunately, this strategy assumed that user problems could

be easily and quickly solved. This was later found to be incorrect.

User information problems were more complex and solving them seemed

less certain. Developers needed more manpower, more time, and new

subsystem components. Essentially, a new set of priorities emerged,

since the early strategy for accomplishments had become less realistic.

Given these problems, the system contractors (NASPO and NIGP)

recognized that a broader perspective was necessary. They envisioned

more gradual change in the user environment and more limited accomplishments. ETIP, however, did not accept a significant change from the

original priority. Eventually, the ETIP view prevailed as the working

strategy, but a basic disagreement over priorities was initiated.
This affected the working relationships within the development team

for the remainder of the project.
5.3.3 Team Personnel Factors

General Description

A second process characteristic developers need to monitor is the

need and availability of skilled personnel to the project. Large scale

system developments can involve a broad range of skills over a considerable time period.

Developers must first ensure that the skills needed in the project

team are obtained. These will range from managerial to technical areas.

Recent systems literature, especially in the behavioral sciences, has

also been emphasizing the need for a mix of interpersonnel and technical

skills. This is a recognition of the organizational problems which new

systems can entail. Developers need to monitor whether the necessary

skills are being applied.

Second, developers must monitor the turnover in personnel that is

likely to occur. Turnover of key personnel can slow progress in a development by consuming time to transfer knowledge between incoming and

outgoing personnel, or by bringing in new perspectives which must be

accommodated with already existing strategies. While some protection

might be made in advance to reduce this problem, developers need to find

ways to minimize the impact of turnovers. This requires close monitoring and early identification.
286

A final area developers should monitor is commitment of personnel

to the project. Commitment of the staff to a project is a predecessor

to the more comprehensive commitments by users. Changes in these commitments, or differences between team members, may be an early indication

of uncertainties in project goals or strategy. They also may be a signal

of forthcoming problems for full scale operation of the system. Developers need to monitor the commitment of team personnel to identify changes

and examine their implications.
5. 3. 3.1 Factor 15 — Skills

Description — The availability of technical and interpersonal skills in project teams as required by system

development tasks.

Discussion

System developments require a range of skills which developers

must ensure are available. At one level, there are the technical

aspects of design which require people trained in specialized areas.

For example, computer system developments would need technical analysts skilled in hardware design. At a second level, there are interpersonal aspects of design which require people trained in understanding and working with people. The importance of both types of skills

is recognized increasingly in the information systems literature,

especially in behavioral and management science sectors (Argyris, 1971;

Miller and Wilier, 1977; Nicholas, 1978). Development models prepared

by authors in these areas place heavy emphasis on the interpersonal relationship between developers and users (see Chapter 3, section 3. 3. 3.3).

While it is useful to match people to jobs, developers must be

wary of utilizing personnel who are too highly specialized. Systems

people should have a mix of both technical and interpersonal skills

so that they can more effectively handle the complex, intertwined

problems that developments involve. Developments can fail because

people on the development team have an uneven awareness of the differ-
288

ent aspects of these problems. For example (Mumford et al. , 1978,

p. 235; Thompson, 1978b):

• Technically oriented system analysts and designers may

have little knowledge or interest in human engineering issues.

• Managers, on the other hand, may be minimally trained on technical matters and prefer to concentrate on budgets and

schedules. They may be reluctant to intervene in design

decisions.

Developers need to examine whether the requirements of system

tasks are being met by people with the appropriate skills and backgrounds to handle them. Contracting for these skills may be necessary, and this may present an additional number of problems developers

should consider. For example, contracting requires developers to

design a procurement procedure that specifies the kinds of skills

needed. In addition, contracting for support results in a new, outside team that requires integration into the user organization.

ETIP Examples

Skill problems were prevalent in the state and local procurement

development discussed in Chapter 2. The initial strategy used in

the project was to identify user information problems, locate appropriate information sources, and then transfer the information. It

was assumed that little or no modification of the information would

be necessary, since planners expected that it already existed in other

more advanced state and local user agencies. Developers, therefore,

initally placed less emphasis on technical skills than on marketing
and user contact skills.

Later, however, it was learned that user problems were much more

complex and would require original technical work if they were to

be handled. This expertise was not easily available to the project,

either by contract or outright hiring and progress was delayed as

a result. It can be argued that the complexity of user problems might

have been identified sooner if technical expertise had been more preva

lent on the development team in the beginning.

It is also interesting to note that it was well known in the

project that the contractors had little experience in systems development. Even the contractors admitted this in the early phases. As

a consequence, little broad thinking was done on user problems, overall system design, or different design options. The lack of systematic analysis was mentioned several times by ETIP, but few improvements

occurred in the contractors. When project resources were dwindling

and few accomplishments were on record, developers quickly dropped

any further system considerations in favor of emphasizing a more limit

ed number of specific projects. These were no longer tied to any

overall system design strategy.
290

5. 3 *3. 2 Factor 16 — Turnover

Description — The change of personnel involved with

project teams.

Discussion

A major problem for a development team is turnover in its personnel (Ackoff, I960; Fronk, 1978; Krasnican, 1971, pp. 5^-55; Patterson,

1977; Thompson, 1978b; p. 329). Important knowledge can be lost in

key project areas that would ensure continuity. Even in cases where

no knowledge is lost, newcomers will need time to familiarize themselves with a project. This can cause delays. If they also bring

a different perspective to the design or the development process,

then conflicts with remaining personnel may arise. In addition, in

cases where users are key team members, turnover may affect the user

organization as well as the team (see also User Commitment Characteristics, section 5.4). Support for the system may be affected.

Turnover in the project team is likely, however. Large scale

system developments can easily take a length of time which exceeds

the tenure of some people, especially when there are long delays (Ruth,

1978, p. 38 ). It is also possible that the more experienced systems

people may leave when the interesting problems have been solved.

For example, as development reaches the operational/maintenance phase,

project activities may be relegated to lower level staff (Chervany,
1978, p. 177). Well experienced or qualified people are also likely

to be in demand for similar projects elsewhere, and thus may be the

first to leave (Ruth, 1978, p. 38).

Developers might prevent some turnover by initially obtaining

long term commitments from key team members who would represent a

significant loss to the project (Ruth, 1978, p. 38). If contractors

are involved in the project, this is often done through the use of

key personnel clauses in the contract.

Changes are, however, inevitable and developers need also to

examine ways to minimize their impacts (Alter and Ginzberg, 1978,

pp. 26-28). Several actions may be taken. Developers can establish

an executive management team, and other teams for that matter, such

that key tasks and responsibilities are shared among several people

(Ruth, 1978, p. 38). This may help avoid the problem of having important project knowledge concentrated in one person. It may also minimize the uniqueness of any one job. Thus the skills needed in tasks

become available in a team instead of depending on hard to replace,

broadly trained individuals.

Another method for use during a project is for developers to

continuously encourage people to remain throughout the development.

Continuity in a project can be more easily ensured if the same team

remains in all phases, design through implementation (Edelstein and

Melnyk, 1977; Ginzberg, 1978a, p. 60; Comptroller General, 1978,
292

p. 8; Radnor et al., 1970).

Also during a project, developers should monitor task structure

in relation to how personnel are assigned. Team members with numerous

assignments over several tasks can have more of an impact on a project

than those who are committed to a smaller number of areas. Developers

may want to reduce broad assignments for project personnel. This

may of course be a difficult strategy to use for top managers of the

development, and other measures may have to be considered.

ETIP Examples

Some turnover occurred in the three ETIP cases reported in Chapter Chapter 2. However, it was difficult to analyze their impacts

and the effect of the changes are thus unclear. No examples have

been selected for presentation here.
5. 3 -3. 3 Factor 17 — Commitment

Description — The team member support for and implementation of the goals, strategies, and tactics of the

system development.

Discussion

One of the major themes in this dissertation is that user commitment to a system is a sequence of actions over time. It begins with

the initial idea for a system and continues as long as the system

is in operation. A number of different factors can be used to monitor

commitment, and these are more broadly covered in the following sections on user commitment characteristics (section 5.4).

A portion of the commitment process of concern here is the commitment of personnel on the development team to the project. Commitment

of project team members to the goals, strategies, and tactics of the

development is an important predecessor to the eventual full user

commitment to the system. A lack of personal commitment may result

in key staff turnovers (see preceding section), or be evidence of

a much broader problem in the home organization. Gaps in commitment

between team members may be a source of confusion and uncertainty

which slows progress. Gaps between team members and their home organizations may have the same effect. Developers (team members) need

to be sensitive to the commitments of personnel in the project. Shifts

or gaps need to be detected early and their broader implications handled
294

in line with project goals.

Monitoring commitment to the project involves a number of indicators applicable to each of the different groups involved. Projects

of interest in this dissertation usually involve several groups who

have few organizational ties. First, members of the user organization

usually manage the project and have other staff participating in speci

fic tasks. Second, other outside groups, such as ETIP and system

support contractors, also have a central role on the team. All of

these groups are referred to here as developers and commitment characteristics are relevant for each.

If a project involves numerous uncertainties and is considered

a novel endeavor, top quality people from each group are likely to

be involved on the team. This includes both managerial and technical

personnel. Contractors may commit their more experienced personnel

initially if they envision the experience to be important for their

future growth (Thompson, 1976, p. 43). Similarly, a user organization

may view the system as a key resource for their work and thus may

want their best people involved (Katch, 1978, p. 54). The initial

lack of or later turnover in key personnel may be a signal that the

project is off-target, or becoming increasingly isolated from the

major concerns of the various home organizations involved. It also

may simply be a sign of personal disagreement over strategy. In any

case, developers should identify these conditions and openly consider

the implications of any commitment changes in key personnel.
A related indicator of project commitment is top level management

support from the home organizations. Most of the systems literature

suggests that this is one of the more important factors related to

successful development (Boland, 1978; Comptroller General, 1975, p. 8;

Krasnican, 1971; Zand and Sorenson, 1975, p. 545). Large scale

systems can have wide ranging impacts on an organization and can involve numerous, previously unrelated people. This may be the case

both within, and between, the home organizations. Top level management support in each organization helps ensure that the broad range

of personnel involved work together (Ackoff, I960, p. 262; Comptroller General, 1978, pp. 9-10). Support at a high level may also act

as an incentive to the various personnel to get involved (Lucas, 1975b,

p. 34). Developers should monitor the extent of high level management

participation and initiative as a sign of project commitment (Katch,

1978, p. 54).

A third indicator for monitoring commitment is the overall size

of the project relative to similar work or competing work in the home

organizations (Thompson, 1976, p. 43). The home organizations may

have several on-going projects which are competing for the same personnel. Personnel may become too broadly committed to a number of different projects which reduces their effectiveness to the system development. Developers need to monitor the position of the development

relative to other work in the home organizations and gauge changing

commitments of personnel accordingly.
A final area to monitor is the difference between commitments

across the groups involved. For the case of systems developed by

several different organizations, one group may maintain commitment

to the project, while it decreases in others. Differences over project goals, strategy, or tactics may cause changes in personnel with

the result that the team then exhibits a cross section of talent,

perspectives, or status. While a cross section like this is not inher

ently a problem, it may create a mismatch of personnel that leads

to conflicts.

ETIP Examples

A good example of the project commitment problem is exhibited

in the state and local procurement system case. This project began

with a fairly strong commitment across all of the organizations involved. FSS was a key supporter of the project plans and expected

to participate in the development and use system outputs. ETIP commit

ted a high level of funding to the project. Top level people in the

two contractors (NASPO and NIGP) were involved in the planning, saw

the project as a means for further strengthening their ties to their

constituencies (procurement officials, the end users), and committed

much of their time to development. Support also extended across a

broad range of the eventual state and local users.

Later points in the project did not exhibit this high level of

interest, however. Federal participation was low, and motivating
participation at the state and local levels was difficult. The organizaticnal and technical difficulties involved in developing the system

caused a split between ETIP and the contractors over the best approach

to reach project goals (see Chapter 2 and Task Structure above, section 5.3.2). The contractors maintained a commitment to project ideas

but changed their view of the strategy needed. They became convinced

that slower, more gradual change was needed at the state and local

levels before a system would be accepted as a routine mechanism for

solving information problems. ETIP, on the other hand, continued

its commitment to the original strategy, which involved achieving

specific changes with users. ETIP also downplayed more of the "systems"

aspects of the project in order to obtain results. The project was

eventually terminated without much accomplishment in either area.

In retrospect, it appears that the differences within the project

team should have been examined earlier by developers. The changes

in participation were an early sign of decreasing interest in the

project. Earlier attention to the problem would have enabled developers to consider other goals and strategies more attractive to the

team. Unfortunately, the awareness of differences came fairly late

in the project and probably made it more difficult to adjust the development and successfully complete the project.
298

5.3.^ Project Control Factors

General Description

One factor having a significant effect on a system development

is the process used by developers to control the project. Large scale

system developments are complex undertakings which involve numerous

activities, problems, technical issues, opportunities, etc. They can

be further complicated in situations where more than one organization

is involved. In this case, new relationships will have to be established

and the activities of unrelated organizations coordinated. Developers

will need to examine in advance how to control the development and delegate responsibilities. They will also need to monitor the control processes used so that revisions can be made if they prove to be ineffective for accomplishing project objectives.

Three categories of process control methods will be discussed

in this section. First, the organization of the control structure as

to teams and responsibilities will be examined. Developers need to

establish a management structure which matches the complexity of the

development and the distribution of responsibility. Second, the use

of decision points and milestones will be reviewed. Developers can

use specific points in time or the accomplishment of selected tasks

as control points where progress can be reviewed, problems can be identified, or approvals for further work can be obtained. Finally, developers can use reports and reviews to monitor work, document and dis-
cuss complex problems, or as a means of disseminating information about

the project to relevant outside stakeholders who may have influence

over the development.
5.3* 4 •! Factor 18 — Organization and Responsibilities

Description — The structure of project teams and respon

sibilities in the system development.

Discussion

A key part of project control is the team structure used by developers to manage and conduct the project. Part of the structure

should include a central authority which can coordinate and manage

the numerous organizations and people involved. This central entity

also needs to develop and refine objectives and strategies as the

development proceeds. Another part of the structure will involve

the distribution of responsibilities to the personnel actually performing the work. Day-to-day control of project activities will be located at lower management levels. Much of the technical problem solving and delineation of strategy in a large complex effort should also

be conducted at these levels. Developers will need to examine early

in a project the kind of structure they use to conduct and control

the work and then progressively monitor its effectiveness in accomplish

ing project tasks.

Many literature sources, especially in the management and behavioral sciences, advocate the use of a small, central team for managing a systems project. This team controls development overall and

reports to higher level decisionmakers in the organizations supporting
the project. A common problem for the team is who has final authority. In projects where only one organization is involved, such as

with an in-house development, the team should probably consist of

both users and designers with users retaining ultimate control (Lucas,

1975b, p. 112; Lucas, 1978b, p.43). This would also help build commit

ments from users. The problem is more complex when multiple organizations are involved, however. For example developers may elect to

use outside contract support to conduct the project, or a third party,

such as ETIP, may be providing the resources for development. In

these cases, the theme of user control may still apply, but project

control may be more shared among the participants. For example, basic

project control might be the responsibility of a single contractor,

who must, of course, then report to and meet the needs of the contract

ing user. The central development team may then consist mostly or

completely of personnel outside of the user organization, while the

users themselves utilize their own team to control the contractors.

Thus in the end, users do retain control of the development, but a

large portion of the responsibilities are located elsewhere.

This problem of control in multi-organization developments can

be a critical one for developers. Generally, it is probably advantageous in large scale developments to have central authority (management) located in one organization. For example, in military system

developments, a number of different arrangements have been tried

(Livingston, 1959):
302

• Central control in the government, with numerous associate

prime contractors.

• Central control in one prime contractor, with numerous as- sociate prime contractors.

• Team contracting, where one contractor assumes the lead

role among a group of contractors committed to work as a team.

In each of these cases, one organization assumes the lead role of

control and the others recognize their authority. Similar arrangements could be possible even within a one organization development

where numerous in-house divisions are involved. Developers, and their

respective home organizations, will have to decide in advance where

the locus of control will reside and then use formal agreements, such

as contracts or interagency agreements, to establish the structure.

The effectiveness of the agreed upon structure will also have to be

monitored during development to ensure that it works. Formal agreements shouldn't act as a barrier to revision of the structure if developers find that objectives are not being met (Gordon, 1980).

An additional, related problem is whether developers utilize

new teams or existing ones to form the working level structure. Developers may choose to form new teams to conduct the work by drawing

on personnel from existing organizational units. These teams can

be designed to include the appropriate skills and levels of manpower

needed to perform a task. They can be mobilized and demobilized as

required (Flagle et al., I960, pp. 102-103). On the other hand, these

teams will be new to an organization and people will have to be shift-
ed around (Chestnut, 1967, p. 39; Thompson, 1976, p. 44). The project

will have to be placed at a high enough level so that transfers can

be accomplished (Flagle et al., I960, pp. 102-103). There may be

a transition period as teams are formed and new relationships established. Personnel may not be available, especially if their loss

would cause significant disruptions to their home units.

An alternative to the project structure is to rely on already

existing groups or structures. In this case, project tasks would

be assigned to the units who would then decide how to perform the

task and staff up for the work. While this avoids the problems in

creating a new project team, several others can emerge. First, existing groups may not have the needed skills or manpower to perform the

task. They may have to rely on liaison with other existing groups

in order to supplement their capabilities. This could create additional levels of authority and control which are far removed and possibly less affected by the central project authority. Already existing

units will also have their own well established agendas which will

compete for staff time. Staff loyalties will likely reside in their

home unit. Developers may find a structure such as this too cumbersome for most project tasks. However, it may be useful for some of

them and be workable in combination with other newly created teams.

Whichever approach is chosen, developers will have to examine

how responsibilities are delegated to the working level teams. Centralized control of all project activities will likely be impossible.
304

and for that matter, inefficient and unnecessary. Numerous problems

and decisions can be delegated to lower level working groups where

they can be effectively handled as they arise (Peck and Scherer, 1962,

p. 455). It will be advantageous overall if working level groups

can gain the ability to recognize problems and opportunities on their

own and then have the flexibility to handle them as appropriate for

project objectives. Large scale, complex system developments will

involve too many problem areas for a central team to handle. In addition, the time to solve them through the central authority will slow

progress and reduce the likelihood for creativity needed in complex

problem solving.

It should be recognized, however, that some problems (or opportunities) may arise which fall outside the organizational lines chosen

by the developers. Developers should consider the need for ombudsmen

who can be assigned to these unique problems (Lucas, 1974b, pp. 30-

91). These individuals might report directly to the central management team.

ETIP Examples

The state and local procurement project illustrates a number

of control problems that systems projects can have, especially with

multiple organizations. The development involved several different

groups: ETIP, the two contractors (NASPO and NIGP), federal, state,

and local agencies. The two contractors were responsible for develop-
ing the system and were to eventually provide its home. ETIP provided

the resources to fund the development and expected to supply technical

guidance as needed. Another federal agency was interested in the

system and was to provide technical materials and support if requested

by developers. However, there was no formal agreement with the agency

on its role. Finally, state and local agencies, the system users,

were to provide manpower and test sites for development activities.

The central control in the project was located within the two

contractors. Each contractor was responsible for delineating the

specifics of project goals and designing activities accordingly.

This had the advantage of giving some user control to the project,

since the two contractors were professional organizations representing

users.

Since the two contractors began at different times, each contractor began with its own central team. These were loosely coordinated

during phase 1 due to the similarity of objectives. This autonomy

was altered in phase 2, however, when concern increased over the slowness of progress. ETIP then chose to bring the two contractors into

a closer working relationship where they could share technical materials and resources. ETIP also assumed a more active role in guiding

the contractors than had originally been planned. It would appear

that central project control shifted some to ETIP.

At the working levels, both contractors formed task forces of
306

volunteers from state and local agencies. These task forces were

to develop information packages which would be pilot system outputs

for use by procurement officials. Volunteers came from agencies with

an interest in the development of a particular package. This mechanism was itself a pilot test of the process developers expected to

institutionalize into the system. NASPO chose to have the task forces

run on their own with minimal technical guidance from the central

team. NIGP chose to control the groups more closely by hiring an

in-house technical coordinator.

As described in Chapter 2 , these teams encountered serious difficulties in completing their tasks. User problems were more complex

than originally thought. NASPO, NIGP, and ETIP had little control

over the task force volunteers and the project was competing for the

time of busy officials whose priorities were still with their home

agencies. Accelerating task force progress was a problem and acquiring more manpower from other organizations, especially those with

a background in the relevant technical problems, proved to be difficult. As a result, ETIP had to again assume a more active controlling role to attempt to solve this problem. When additional manpower

was not forthcoming, ETIP elected to reduce the extent of contractor

activities so as to increase the manpower and resources on a few selected task forces.

Very similar manpower problems also emerged in each project concerning the conduct of system design tasks. Manpower experienced
307

in the development of large scale systems was very limited in both

contractors. In addition, most of the available manpower was directed

toward managing and accelerating the activities of the volunteer task

forces. ETIP first became concerned about the delays in systems tasks

during the middle of phase 1. ETIP again chose to temporarily take

lead control over the project and start the design activities. Eventually, a pilot design was produced for the information items to be

included in the system. ETIP then returned to its support role.

ETIP reentered project control again in phase 2 when continuing

delays in the task forces made further progress uncertain. At this

point, ETIP elected to concentrate most project attention on the task

forces and also began to have a more active role in decisionmaking.

The two contractors did not completely agree with this narrow focus,

however. They believed more fundamental, incremental changes were

needed before task force outputs would be useful. This split in approach may have placed even more of the project control within ETIP

instead of the contractors.

In retrospect, it would appear that one of the central problems

of the project was the need for more manpower and technical expertise

to be under the direct control of developers. Developers had little

flexibility in their working teams to accommodate the technical problems due to the unavailability of their volunteer members. Developers

also found expertise difficult to acquire elsewhere. More in-house
308

staff with full commitments to the project probably could have accelerated the work and helped to identify the user problems more quickly.

This might have altered project strategy much earlier.
5.3.^*2 Factor 19 — Decision Points and Milestones

Description — The structure of specific events created

by project managers to recognize or review progress and

decide future courses of action.

Discussion

Large scale system developments involve numerous activities,

problems, and opportunities which need some level of ongoing recognition and approval from the central project team. For example, several

occasional high level decisions will be needed to select from among

the various design options and strategies available and then to allocate resources accordingly. Decisions like these may require formal

acceptance by all of the different organizations involved in the project. Conversely, numerous lower level decisions on specific tasks

and problems arising within project phases will also be needed. While

these may not require recognition by all of the project participants,

they might provide useful milestones for gauging progress. Developers

first need to examine in advance where major and minor decision points

should be placed and then actively monitor their occurrence.

For the higher level issues, decision points are often placed

at the end of phases which denote changes in the development. For

example, many system development models divide a project into phases

such as planning, design, testing, and operation/maintenance. This

structure implies an orderly transition from designs on paper to hard-
310

ware assembly for all system components. Decision points are located

at the end of each phase to mark the conclusion of the activity and

to select the strategy for the next phase.

The exact placement of major decision points is fairly arbitrary,

however. Activities implied by the name given to a particular phase

probably recur throughout the development of a system, particularly

if the progress on different subsystems varies or if cycles are made

to refine earlier designs in light of new information about the user

(Weiss, 1977, pp. 38-90). For example, planning on one subsystem

may occur while others are being tested. Developers may have to select points at which the development is, in general, making a transition.

The flexibility in placement is not meant to indicate that major

decision points have a minor importance. In developments with significant uncertainties and risks, placing decision points at the end of

phases provides some protection against making commitments too early.

This is more simply called the "fly before buy" strategy (Gordon,

1980, p. 39). Major decision points also provide a means of controlling a project, especially in assuring that organizations centrally

involved in the development have the power of control. For example,

the General Accounting Office (GAO) found that government systems

projects often acquired a momentum of their own and were completed

without any major approvals from users (Comptroller General, 1975,

pp. 8-9) (see section 3. 3. 3*2 also). Projects moved along according
to original plans and success or failure had little effect on the

transition from one phase to another. The GAO recommended that formal

decision points be included at the end of each major project phase

and that separate budgets for the different project phases also be

used

Major decision points at the end of phases are not the only control points needed in a project. Developers will also need to emphasize sequential approval and acceptance of progress during each phase

(Thompson, 1976, p. 49; Zand and Sorenson, 1975, p. 545). Problems

and opportunities can be identified more quickly and acted on at a

point when action is needed (Gordon, 1980, p. 34). Reworking, redesign, and argument after the fact can be avoided (Thompson, 1976,

p. 49).

Within each phase, then, developers should select events or milestones as formal, recognized points for monitoring. Milestones can

be either points in time or events (Gordon, 1980, p. 34). They might

represent a series of events or a composite of actions which conclude

with some identifiable accomplishment. The key is for developers

to select milestones which provide insight to project activities and

prediction power for future success or failure. They should be something which are (Gordon, 1980, p. 34):

• Objective, in that they are not subject to widely different

interpretations
• Material, or can be seen directly, and

• Significant, in that their accomplishment has some significance in the project.

A series of milestones can have the additional benefits of leaving

a better documentation trail and improving the decisionmaking conducted periodically at the end of phases.

ETIP Examples

In retrospect, it is difficult to identify the selection, monitor

ing, and use of decision points and milestones in the three ETIP cases

Generally, documentation of actual progress is much briefer than that

found in the original project descriptions, plans, and structures.

A complete record of milestones and major decisions is not available. Thus, it is difficult to directly identify how they were used

by developers for project control.

Nevertheless, some conditions can be implied from the available

project histories and used to illustrate the decision point factor.

For example, the SBA data base system project shows a degree of project control via a series of major decision points and project phases.

The project had six phases: problem definition, preliminary design

and planning, operational planning and full development, test planning, testing, and operation and maintenance. Each phase was marked

by a final report from the contractor and a decision by ETIP and SBA

to proceed to the next stage. The support contractor accomplished
the stages on time and generally demonstrated orderly progress on

the objectives. In addition, plans for subsequent stages were constantly updated at the end of each phase and thus were successively

reapproved by the central development team.

This structure did not appear, however, to have been effectively

used by developers to help control the project and assure success.

By the time developers reached the operational testing phase, the

numerous problems users had with the design slowed further progress

considerably. Test cases from users were very difficult to obtain,

even if the cases were retrospective and thus would not have been

affected by a test. The preliminary manual system design quickly

approved in earlier phases was not, in the end, very marketable to

users. Progress remained delayed until major design changes were

made and the computerized system developed.

Another example is found in the state and local procurement information system project. In this case, two contractors were acquired

to develop a system for state and local procurement officials. The

project was divided into two parts, one for state and one for local

users; each was designed to run approximately three years. Each project had three major decision points built into the strategy and budgeting: (1) planning and staffing, (2) specific experiments to synthesize a system, and (3) institutionalization of the system with users.

The use of more specific milestones in each project was different,

however. In the state level project, specific milestones were developed,
but they changed several times, especially in their specificity.

It appears that they were developed late in the first phase, which

was the planning and staffing phase, and were not used routinely after

that. The structure and terms varied considerably in subsequent month

ly reports. In contrast, a milestone structure does not appear to

have been adopted in the local level case. However, since both projects were essentially consolidated to one during the second experimentation phase, the state level milestones were loosely applied to

the local case.

In each project, the decision points located at the end of each

phase were used to control the allocation of resources to the next

phase. This provided the checkpoint at the end of phase 2 when ETIP

decided to terminate the projects. While the use of these points

helped avoid the problem of unbreakable momentum discussed earlier

in this section, they did cause distractions from the main project

work. This was because all of the groups involved in the development

spent considerable amounts of time preparing information for the decision. This analysis time may have been the result of earlier inactive

monitoring of milestones which could have developed the needed informa

tion base. This would partially demonstrate the value of successive

approval and acceptances during each phase, as mentioned above.

It is unclear, however, what effect the milestones had for project control. In the state level case, specific milestones were avail

able, but there was a delay in their development, and the structure
315

does not appear to have remained consistent throughout the project.

Their application appears uneven since they were used to signal slippages and lack of progress during phase 1, but not during phase 2.

Also, it is unclear whether the progress in phase 1, according to

the milestones, had an effect on the decision to proceed from phase

1 to phase 2.

In the local level case, milestones were not developed during

phase 1, but the project was loosely gauged in phase 2 according to

the general milestones designed for the state level case.

Overall, for both projects, it appears that the milestones provided some insight to progress and problems, and that developers occasionally used them as a background from which to measure progress.

However, a comparison of the problems occurring with the milestones,

interventions from developers, and the use of decision points, indicates that milestones were probably not actively monitored and used

as control points to alter strategy and implementation as the projects

unfolded.
316

5. 3. 4 *3 Factor 20 — Reports and Reviews

Description — The structure of written and oral commu- nication mechanisms used to document and review development progress.

Discussion

Decisionmaking and control of a large scale system development

will involve some level of information transfer between the various

groups involved in a project. Transfer and liaison with other groups

outside of the project may also be needed (Thompson, 1976, pp. 46-

49). Two common mechanisms for processing and transferring information are reports on project activities and periodic reviews of major

technical and organizational level issues. Developers need to structure in advance where these reports and reviews occur over the life

of a project and who may be involved. Developers then need to monitor

whether these mechanisms prove to be sufficient for transferring key

information of the level and at the timing needed to control the project.

Written reports on development activities or issues can be defined at a number of different levels. First, the content of reports

can vary by purpose. Reports might be generated on schedules, costs,

expenditures, designs, strategy, etc. Second, reports can be produced at various times, such as periodically or on a special, one-time

basis. Third, reports can be produced by different groups, for dif-
ferent groups. For example, if the development involves contract

support, there may be reporting required between contractors and the

development team, and also between the development team and higher

level members of the user organization. A key problem is proper identi

fication of the organizational levels (on the team and in the user

organization) at which reports should be directed (Ackoff, I960).

Developers need to consider several factors when designing or

scheduling written reports. First, reports require a significant

amount of time to prepare. They can distract key members of the development team from the priorities of the project. Developers should

keep the number and extent of reports small and monitor whether too

much effort is being diverted to their production.

Secondly, reports may serve purposes beyond project control which

need to be considered when reports are designed. For example, developers may want to document portions of the project for later review

(Thompson, 1976, p. 46). This might be important in areas where 'uncertainties remain and developers choose to forego further work (Krasnican

1971, p. 55); later exploration could be more fruitful. Reports

might also act as marketing documents for the project and the system.

Organizations or institutions outside of the system may have a central

interest in it and developers might choose to disseminate information

to them. As with groups internal to the user organization, a system

may also appear threatening to these outside groups and developers
should consider how reporting might reduce this problem (Thompson,

1976, p. 47).

Thirdly, developers need to consider the appropriate level of

reporting. In some cases, more informal liaison between groups may

serve the purpose of transferring the needed information, while in

other cases more formal communication through channels is needed

(Thompson, 1976, p. 47). This could affect the type of report needed.

Finally, since reports take time to prepare and revise, their

utility in real time project control may be limited. Developers may

need to rely on other means to help control a project (see design

reviews below) and let reports act more as a record of already recognized and approved activity (Thompson, 1976, p. 49).

More direct real time control may be exerted through the use

of face-to-face meetings and discussions which can quickly pinpoint

problems and options. It is common practice in system developments

to use design reviews for the purpose of reviewing general problems,

strategies, and the like, and making changes. For example, military

and system engineering system development models often use major reviews when there is to be a transition from one development phase

to another. These can be called preliminary design reviews, critical

design reviews, first article configuration inspections, demonstration acceptances, etc. (Chestnut, 1967, pp. 37-39, as adapted from

Air Force Systems Command, 1966; Hill, 1970). The basic strategy
of these reviews is to compare progress, either in specifying a design

or actual construction, with previous design plans. Inspections,

for example, mean comparing hardware with the paper design. Demonstrations can mean the review and acceptance of assembled subsystem components in comparison to expected overall system performance specifications. Reviews like these could also be extended to more narrow,

specific problems or subsystems and occur more frequently than once

during a phase. The extent of participation by the central development team might also vary in this case. Developers need to identify

what reviews are useful and who should participate and contribute

to them.

ETIP Examples

It is difficult in retrospect to determine how the reports and

reviews structures of the three ETIP cases were used in project control. Numerous reports were produced in each case, primarily monthly

progress and final reports, and various meetings and reviews occurred.

The effect of these activities on decisionmaking and strategy is not

easily traced. (Part of the problem is due to the fact that the case

studies were conducted much before this section was written.)

However, the state and local procurement project illustrates

a structure of reports and some of the issues of reports and reviews.

In this case, reports were mostly produced by the two contractors

involved in the project. Several different deliverables were sped-
320

fied in the contracts: monthly progress reports, quarterly reviews,

special reports, final reports for each phase, and briefings. In

the monthly reports, the contractors were to detail progress, problems, activities, contacts, and whether ETIP assistance was needed.

Quarterly reports were to be more general statements on progress that

would provide overall perspective on project objectives and strategies. Final reports were to be produced before the end of each phase

and were to be used to support the decision points on whether to move

into the next phase. Generally, this structure was followed in the

projects.

It appears in retrospect that the monthly reports were probably

more useful in documenting progress than for controlling project imple

mentation. For example, in the state level project, the structure

and content of monthly reports varied throughout the project. At

first, monthly reports followed the outline specified in the contract

and this was satisfactory to ETIP. Later, when a milestone structure

emerged for project tasks, monthly reports were altered such that

sections were written on each specific milestone. This format was

discontinued after several months and was not immediately replaced.

About one year later, midway into phase 2, monthly reports reemerged

in a format similar to the initial one, except that progress was now

reported by major task only. The structure of the tasks was different

than that developed earlier in phase 1. This format continued until

the project was terminated.
Using these reports to assist in project control would appear,

in retrospect, to be difficult. First, the content of the reports

varied and thus a consistent structure to gauge progress was not available. Second, the detail used in the milestone structure was probably

too cumbersome to effectively monitor progress overall. Too much

information was presented, and this may have been a greater burden

on the contractors than intended. This may have caused the change

back to a more general level. Third, it seems as though the items

in the reports were being recognized at a time long after which they

needed action.

Project reviews probably compensated some for the problems with

the monthly reports, since records of numerous meetings were found.

However, a planned structure of reviews was not found and only a few

major design reviews occurred. For the state level project, one review was held midway during phase 1. At this time, progress on the

major system development objectives appeared to be slow or inadequate,

especially since little attention was given to them in the previous

monthly reports. The review provided the first opportunity for developers to meet and discuss system design and strategy. Available

documents show that developers outlined the contents of the information system they wanted and some of the procedures they expected to

include. This meeting did affect project design in that the problems

arising with users were recognized by changes in the planned information services. Subsequent monthly reports and other letters from

the contractor included these changes. Attention towards these ma-
322

terials does not seem to have continued into phase 2, however.

One other major review occurred midway in phase 2 when an upcoming decision point was recognized by developers as a critical point

for continued project support from ETIP. As discussed in the section

above on Decision Points and Milestones, the groups involved in

the project prepared a new set of materials for this review. It appears that the previous design specifications and other related materials did not have much of a role in these newer reviews. Subsequent documents prepared for the decision were at a general level

and did not include many specifics about system design. It must be

noted, however, that, at this time, ETIP had elected to concentrate

more on individual experiments with users than on developing the system.

From an overall perspective, the state and local projects partially demonstrate the difficulties of controlling a project through

reports without also including more direct contacts in project reviews.

The various significant problems developers encountered with users

had major implications for the expected system design. While these

were recognized at different times in some of the reports and reviews,

these mechanisms appear to have been ineffective since the problems

reoccurred and the basic strategy for the development was only slowly

changed. Developers may have needed to plan and structure more reviews

in advance, with specific attention to overall objectives, in order

to identify the problems earlier and adjust their strategy accordingly.
323

5.3.5 Interaction with the User Environment Factors

General Description

A key part of system development is the interaction developers have

with the user environment. Interaction is needed in order to transfer

information about the user problems to the design team and to progressively develop and transfer the system to users. Interaction is also

beneficial for establishing relationships with users. Their openness

and support will be critical for design and eventual institutionalization.

There are two related perspectives developers should monitor about

interaction. First, there is direct user involvement in the project.

User involvement is one of the main factors found to be important to

project success in the literature (Keen and Morton, 1978, p. 196). Both

the extent and type of their participation are important (Boland, 1978,

p. 897). Enlisting their participation and support early and routinely

will provide the basis for user commitment to the system (see also User

Commitment, section 5.4). The first section below will discuss these

considerations in more detail.

A second perspective for developers is that of contact with the

user organization in general. Isolation from the user environment can

lead to serious problems in the design that can threaten project success.

Extensive contact should be made in order to extract the key user prob-
324

lems around which a system needs to be designed. As a project evolves,

testing designs in the user organization will be critical for verifying

concepts and checking actual performance. Accceptable designs will then

have to be transferred to users by a means which minimizes disruptions

in the organization. Sections on each of these processes — problem

identification, testing, and transfer — are presented below.
325

5. 3 .5.1 Factor 21 — User Involvement

Description — The participation of members of the user organization in the managerial and technical teams

of the project.

Discussion

One of the major factors found to be important in system development

is the involvement of users (Keen and Morton, 1978, p. 196). Projects

conducted in isolation from users or without the active participation

of users have often resulted in designs which do not meet users needs

and gain little support for institutionalization. Developers need to

identify areas where user participation is required or beneficial and

monitor project tasks to assure that participation is solicited and utilized. Special attention should be given to ic‘^tifying key users who

can articulate the demand for the system in the organization and whose

support may influence the participation of others (Lucas, 1978b; Thompson,

1978b, p. 332) (see also System Champions, section 5. 4. 3.1).

One approach to user involvement is to include users on project

teams, both at the managerial and working levels. This provides a

mechanism for user control of the project and a basis for eventual

ownership of the system (Lucas, 1978b). It also helps ensure access

to the organization for the collection of information needed by design

teams (Nay, 1973, p. 121). Soliciting involvement at this level of the
326

project may be unnecessary in cases where developments are already conducted by users within their own organization.

Developers should also consider involving users from different

levels of the organization. Managers and line groups may have different perspectives on the information problems and the appropriate

system designs (Flaherty, 1980, p. 223). These gaps may hinder progress unless they are resolved (see also Boundary Factors, section

5.2.3). A formal user group composed of users with different perspectives may assist developers in building the needed consensus and momentum (Comptroller General, 1978).

Obtaining user involvement may be a difficult problem in itself,

however. Users may not have the time to be actively involved due

to the press of other business in the organization. It also may be

unclear who should be involved, particularly if the system is a new

entity for the organization and the final home unknown (Thompson,

1978b, p. 332). Early participation may be tentative and weak until

more is known about project objectives and relationships between developers and users are established. As mentioned above, if there

is little consensus on the need for a system, users may also resist

involvement and continue to support other systems (Ruth, 1978).

Close involvement may also have some risks which developers should

consider in soliciting participation. Early project stages are likely

to be characterized by exploration, uncertainty, and shifting concepts.
utilizing the limited time of users at this stage may be inefficient

until clearer tasks emerge. In addition, high levels of involvement may

raise expectations about the system. Difficulties or delays may lead

to higher disappointments later if these initial expectations are

not met (Lucas, 1976, p. 68).

ETIP Examples

The SBA data base system project demonstrates some of these user

involvement factors. In this case, SBA contracted the development

work to a consulting organization. While the contractors performed

some preliminary studies of user problems, they essentially conducted

most of the initial design work outside of the user environment.

Their strategy was to first develop a system which could then be tested with users for an extended period. These tests would depend on

heavy user involvement.

Developers found, however, that the system was not attractive

to users. Some of their key problems were not solved by using the

system. Also, since most users were very busy, it was difficult to

enlist their participation in tests. The SBA central office, which

controlled the development, attempted to improve participation by

sending a letter to supervisors. This had little effect in the end

because field users did not have a close relationship with the central

agency decisionmakers.
328

It would appear that development might have benefited from earlier

participation by users. The key problems could have been uncovered

sooner and integrated into design considerations before significant

investments in time and resources were made. In addition, developers

might have attempted to formulate an advisory group composed of both

field level and central office personnel. This group might have highlighted the different perspectives on the problems and resulted in

the identification of common objectives. This would have probably

assured more participation in the testing phase where developers depended heavily on user acceptance and support.
329

5. 3. 5. 2 Factor 22 — Problem Identification

Description — The overall amount of attention to and the

amount of contact with the user organization in the definition of user problems.

Discussion

One of the more important parts of the development process is

problem identification (Comptroller General, 1975; Hall, 1962, pp. 7-11;

Keen and Morton, 1976, p. 196; Lucas, 1978b, p. 51; Office of

Federal Procurement Policy, 1976). Information problems are likely

to be unstructured and complex. They will probably be closely intertwined with other problems in the user organization. Early definitions of the problem may be too general or simplistic and, in any

case, will probably change as more is learned about the user environment and design options. Developers shouldn’t necessarily expect

to achieve early understanding of the key problems as many may be

uncovered later. Nevertheless, project success will depend on how

well the problems are defined and used to design a system. Attention

to problem definition should be given early and routinely throughout

the project.

A systematic approach to problem identification will benefit

the development (Bostrom and Heinen, 1977). Emphasis should be placed

on learning about the user environment from a number of different

perspectives: how it works, goals, decisionmaking styles, decision
330

processes, information flows, other information systems, etc. (Lucas,

1978b, p. 51; Mason and Mitroff, 1973). The goals and expectations

for a new system would also be an important item to include here.

Developers might start their analysis at the highest decisionmaking

levels and work into the organization. Also, concurrently, they can

examine the lower working levels of the organization and move up decisionmaking chains. The perspectives on the information problem

may vary considerably across the different organizational levels (see

Boundary Factors, section 5.2.3). In addition, adapting a system

to the user will depend on accurate knowledge of these areas (see

Adaptation Factors, section 5.2.4).

Problem identification should also be extended to the external

environment of the user. The user organization will probably have

relationships with outside groups that can have significant effects

on the organization itself (Ein-Dor and Segev, 1978, p. 1073). For

example, government agencies have budgetary relationships with the

Office of Management and Budget. Developers should examine and monitor these linkages and determine how they might affect system design

(see Boundary Factors, section 5.2.3). It would also be beneficial

to determine what information about the system these groups might

need during the development (see also Reports and Reviews, section

5.2.2.3)

As information is gathered from these areas, it may become useful

to organize it by constructing some models (Gorry and Morton, 1971).
331

At first, these might be descriptive models about what is happening:

the players, the decisions, and the information needs (King and Cleland,

1975; Munro, 1978). It may take several feedback cycles with users

in order to obtain accurate models. Later, a second set of normative

models can be developed to describe the decision processes users would

like to have. These models might include changes in the organization

itself as well as new mechanisms. Descriptive and normative models can

then be compared to help identify key information problem areas.

Developers should also consider how the information they collect

during problem identification can be used to help the organization directly (Thompson, 1976, p. 55). A system development may present one of

the first times the organization has been closely examined, particularly

by outsiders. Developers may uncover many factors about the organization

that were not previously known (Argyris, 1971; Krasnican, 1971). Detailed feedback to users about current processes may identify improvements which can be made immediately without proceeding with a new design

(Ackoff, 1967 , pp. B-153 - B-I56 ). Close study may also reveal widely

varying knowledge or perspectives on organizational problems, including

the information problem of concern (Flaherty, 1980, p. 224). It may

be important to project success to reduce these differences. User recognition of the need for change has been found to be a key factor in successful development (Zand and Sorenson, 1975, p. 545). Developers should

spend time communicating and selling the problem to the user (Moore and

Byrd, 1977)
332

Spending time on problem definition with users may also have additional benefits besides locating the need for a system. It will begin

to set a precedence for close user contact and involvement with the development. Some users may not understand what a systems project is or

will have significant anxieties over how it will change their jobs (Argyris,

1971 , pp. B-289 “ B-291). They may resist providing information or access to their programs. Similarly, developers may be uncertain of what

will happen or be inexperienced in what to do. Close contact betweeen

the two groups can help reduce these problems by promoting a mutual understanding (Boland, 1978; Churchman and Schainblatt, 1965). For those

users not directly involved, the perception of close contact may be

important for their support (Lucas, 1975b, p. 3^)-

Other benefits might include the following:

• Establishing a language about problems and designs which users

and developers understand (Zimmerman, 1977, p. I 88 ).

• Identifying key underlying assumptions in problem definitions

which need examination (Mitroff, 1972).

• Discovering areas where resistance to change may be great and

organizational politics important (Keen and Morton, 1978).

• Starting to build user commitment to the system.

ETIP Examples

The importance of problem identification with users was demonstrated

in the ETIP project on the SBA data base system. The project was aimed

at the general problem of increasing the number of government R&D con-
tracts set-aside for small businesses. SBA field officials were responsible for identifying potential set-asides and justifying them

to government agencies.

Developers elected to concentrate on the portion of the problem

related to matching contracts with firms. A system was developed

to categorize the R&D work described in a contract work statement

and then to acquire information from potential firms on the same items.

Firm capabilities or experiences could then be analyzed in detail

as to whether they were similar to the needs of the work statement.

A good match would demonstrate that small firms were capable of performing the work, thus justifying a set-aside to them.

When developers reached the field testing stage, however, they

found that the system was not attracting much interest from users.

Developers had great difficulty obtaining cases, real-time or retrospective, on which to try the design. Users complained that the real

problem they needed help on was simply to find firms whom they could

contact. Matching firms and contracts was something they felt depended

on their professional expertise, rather than simple, easily mechanized

procedures. As discussed in the case history (see Chapter 2), further

progress was delayed until developers produced a system design which

addressed the source list problem.

In retrospect, it would appear that developers did not maintain

an early close relationship to users that might have prevented the
334

subsequent design problems. Even though developers had been in the

field early to collect data on what was happening, the source location

problem was not apparently discovered. It wasn't until developers

brought the system to field workshops to introduce it and collect

test cases, that the more pressing user problems were discovered.

This was not easily compensated for by the developers since many of

the project resources had been committed to the design already and

the remaining strategy depended heavily on quick user acceptance.

Early, more intensive contact and iterations with users might have

prevented these problems and allowed developers more flexibility to

alter their strategy and designs.
5. 3 -5. 3 Factor 23 — Testing

Description — The amount of concept and design testing

conducted in the user environment.

Discussion

As more is learned about user information problems and the user

environment, concepts and designs for the system will emerge. Early

designs may be formally or informally tested by developers within

the project before they are presented to users. Also, user involvement on

design teams may offer a chance to collect early user reactions to design

options without having to make presentations to the user organization.

Eventually, it may become necessary to directly test designs in the user

environment as a means of substantiating design expectations or claims.

Testing in the user environment may also be important and beneficial

f or

• Competitive demonstration.

Several competing proposals for a system design may emerge

in the project and testing in the user environment may be

critical for selecting among them.

• User involvement.

Testing in the user environment can give users something

to react to (Zimmerman, 1977, p. 190), especially those users

not directly involved in project teams. Their knowledge and

opinions may be critical factors for system designs. Participation in a low risk test may be an effective means of obtaining their ideas and support.
336

• Commitment.

Favorable results from tests may be one important factor which

eventually results in full user commitment to the system. In

addition, opportunities may arise for early use of the system

while it is still under development. These may provide an early payoff which should be taken advantage of.

Developers should establish an active program of testing with users and

continue it as long as necessary in order to resolve uncertainties in

design (Gordon, 1980, p. 35).

There are several different levels of testing in the user environment which might be considered by developers. One is to use models on

paper or dummy mock-ups of the system and/or its products (Fronk, 1978;

Lucas, 1978b, p. 44). The earliest of these might be descriptive and

normative models of the user decision processes which evolve in problem

identification activities (see above section). Rapid feedback to users

of information like this can further educate users as to what is needed

as well as provide a test to determine whether developers are understanding their needs (Fronk, 1978; Lucas, 1978b, p. 44).

At another level, working models of subsystems might be constructed

and pilot tested under various conditions with users. Different parts

of the overall design can be tested by themselves and at the appropriate

stage of development. For example, developers might choose to focus

on a particular decision problem or one facet of it and then produce

a working model to test their design concepts for this one problem.
other subsystems might benefit from the results of tests like this or

be developed independently later.

Selecting particular problems and developing simple and limited

subsystems independently of each other can provide certain benefits to

system developers. For example:

• An incremental, evolutionary approach (Davis, 1974, pp. 409-

410; Lucas, 1978b) reduces the degree of risk involved. Developers don't have to commit themselves early to one design

concept or an overall design plan.

• Problems or failures in design can be localized and minimized

(Davis, 1974, p. 408).

• A track record of modest successes may increase the likelihood

of overall success (Edelstein and Melnyk, 1977; Ein-Dor and

Segev, 1978, p. 1073).

• Tests involving designs that make small, incremental changes

in procedures may entail less user resistance and open the

way for a new approach (Thompson, 1978b).

Complex functions and their testing can be added later when the different, specific facets of a process have each been individually studied

and understood.

Finally, as many of the system components become developed, a prototype system can be formulated and tested with users. A prototype design

would be expected to include most or all of the functions developers

had found to be needed by users. It would also include the various interconnections between subsystems or between the system and outside groups.

The prototype design would provide for a full scale system test and

check the emerging design against the original performance objectives.
338

An important part of testing at of any of these levels will be the

development of user criteria to evaluate the performance of designs.

Developers will need to identify in advance and progressively refine

the criteria users prefer to apply. Besides inquiring directly about

these with system users, developers might also consider examining organizational review processes which already exist. Also, there may be criteria originating from outside the user organization and developers may

need to interact with these groups. Developers should also consider

how the criteria and procedures used for one test can be used for future

tests and updating. Some of them may be appropriate for institutionalization within the system for routine use after development is

completed (see also Evaluation and Updating in section 5. 2.4.4).

ETIP Examples

All of the ETIP system development cases included some form of testing in the user environment. In the SBA data base system and the PBS

planning system cases, testing was most prevalent in the final stages

where developers entered operational testing phases. Here the objective

was to utilize the new system in both retrospective and real time decision problems where the system was needed.

These tests proved to be important to overall strategy. In the

SBA case, developers found mixed feelings among the potential users.

Many considered the system to be aimed at the wrong problem and resisted

participating in any testing. Others provided some retrospective cases
for developers, but were not attracted enough to attempt a real time

test. These reactions were a major problem for developers since they

had planned to test the system for one year using both retrospective

and real time cases. The lack of participation left them few means to

establish a track record of success which they could use to market the

system to users. This was critical to developers since their own inter

nal tests did not appear to be sufficient in themselves to attract wide

spread use. Given these kinds of problems, it might be argued that developers delayed too long before initiating active testing in the field

with at least some pieces of the design.

In the PBS planning system case, testing demonstrated both problems and successes which were beneficial to project strategy. Developers were able to conduct tests on both retrospective and real time

cases. Tests showed that users were not very interested in using

the main analysis features the new system offered. However, the tests

did show that some limited portions of the analysis capabilities were

attractive to users and could easily become routine parts of user

processes. This had not been expected earlier by developers. While

it represented only a very modest success for the project, it did

show promise for institutionalization at the end of the project.

It was also possible that the parts of the system not being used might

later become operational. It is hard to measure whether earlier testing directly with users would have achieved this same end result.
340

The state and local procurement system case had a slightly different

approach to testing than did either of the other two cases. In this

project, developers chose to test small models of the expected system

process early with users. This was to help them test their ideas and

gain an understanding of user needs. Special, common information problems were identified in early stages of the project and work was started

to develop packages to help solve them. Developers were to then market

the packages to users, along with the system design. Eventually it was

hoped that users would begin to support the system.

This early testing strategy proved to be a good one in that developers learned the extent of user problems through direct contact. They

found that user problems were more complex than originally thought.

However, developers apparently had difficulty translating these

results into an improved development strategy. The same strategy was

maintained throughout the project, with some minor variations, and user

problems continued as before. Eventually, project attention was focused

exclusively on the tests. It may be that too much attention ’/ras placed

on the tests versus the system design, ignoring especially the implications for design based on the lessons being learned. Developers might

have tried to improve their design and then reentered the user environment for more tests. Competitive testing of alternatives, or at least

active consideration of alternatives based on available test results,

might have resulted in a design better matched to the circumstances.
5. 3. 5. 4 Factor 24 — Transfer

Description — The amount of attention devoted to the

transfer of the system to the user environment.

Discussion

A key transition point in a systems project is the transfer of

control and support from the developer to the user. Transfer of the

system to the user is often referred to by several names: implementation, institutionalization, cut-over, or conversion. Conventional

development models usually place these activities in late phases of

the project (Ackoff, 1967; Alexander, 1974; Davis, 1974), while newer

evolutionary models emphasize more gradual transfer of ownership (Lucas,

1978b; Office of Federal Procurement, 1976). These newer models recognize the human and organizational impacts changeover can have and

the need to plan transfer as early as possible. Whichever approach

is taken, developers need to think through the implications of changing to a new system and routinely monitor whether their strategy is

leading to a smooth transition of ownership and full user commitment.

Transfer of a system to the user is an activity which can and

should begin early in a project. As concepts and pieces of the design

evolve during problem identification activities, developers should

consider how the designs would become fully operational and institutionalized within the organization. The differences between the new

design and current practice will help identify possible transition

strategies and key problems (see also Novelty of the Design, section
342

5. 2. 4.3). Asking users directly about transfer related problems would

also be a useful strategy. For example, involving users who will

eventually support and operate the system when it is fully institutionalized may provide insights for the design which would facilitate

its transfer to them (Katch, 1978, p. 59). Similarly, close user

involvement on the design teams will provide a means to transfer concepts. This would help establish a basis for later transfer of the

specific designs.

As designs are detailed, tests may be conducted to verify concepts

and performance. This is another stage in which transfer considerations

should be studied by developers. Tests in the user environment may be

designed to include user participation in using and working the system.

This will provide a temporary means of exploring how to transfer system

components to users from the developer perspective. It will also provide

a low risk means for users to explore the transition to a new system.

Feedback from their perspective will be useful. Developers should be

able to learn lessons about transfer from testing which can be used

later in future tests and final institutionalization.

When it is appropriate or required, the system or individual components will be transferred permanently to users (the turnkey system).

Developers may have the choice of transferring the system gradually component by component, or quickly making the transition from the old to
343

the new system. Several factors can be considered in deciding which

approach to take:

• Abrupt transition.

Complete rapid transition to a new system may be too risky

or unnecessary. If there are user problems needing quick

attention, it may be beneficial to institutionalize key com- ponents as they are available. Radical changeover may create

a number of side impacts with users that only complicate the

transition and start-up periods.

• Backup systems.

It may be beneficial to gradually phase in the new system

while gradually phasing out the old. This will provide some backup or protection should unexpected problems occur. Users

should not be left without any system if delays occur (Ruth,

1978).

• Training.

Users need to be trained to work with a new system. Users

should be involved in developing training programs (Thompson,

1978 b, p. 332 ) and training needs should be a part of design

activities (Lucas, 1975b, p. 112). The availability of

these programs may influence the transfer point.

• Documentation.

Documentation of the design and related matters should be

complete as possible, especially before transfers are made.

Areas where design problems remain should be thoroughly documented so that users can continue development later (Krasnican,

1971, p. 55).

• Management support.

As the point for transferring a system to users approaches,

it is possible that managers (both of the project and the user organization) may start assuming that the system is complete

and its implementation routine. This may cause a lack of attention towards transfer which jeopardizes final stages. For

example, lower level and less experienced staff may be as- signed to transfer tasks and cause implementation failure

(Chervany, 1978, p. 177).
344

When the system has been mostly or completely transferred to

the user, a final stage of withdrawal and termination should begin

for the project team (Kolb and Frohman, 1970). In termination,

developers need to ensure that (Alter and Ginzberg, 1978, p. 24):

• Ownership and control of the system rests with those who must

use and maintain it.

• Necessary new patterns of behavior have become a stable part

of the user's routine.

The basic goal is to re freeze the organization (Lewin, 1952; Schein,

1961 ) by removing the disturbances of change and leaving the organization

in a stable position (Zand and Sorenson, 1975, p. 545). Termination

should probably not be abrupt. For example, developers might continue

to monitor system performance under full user operation and help with

any unusual problems that arise in start up. Also, developers should

ensure that evaluation and updating functions become operational so that

users learn to revise the design themselves (see also Evaluation and

Updating, section 5. 2. 4. 4).

ETIP Examples

The importance of transfer varied in the three system development

cases. In the procurement information and PBS planning system projects, the systems were developed under the control of the eventual

system owners. Direct users, however, were to be officials outside

of the home organizations. Also, to some degree, the systems depended
345

on outside groups for manpower, data, and technical support. It appears that neither project employed a transfer strategy to develop

routine, on-going relationships with these groups. In the PBS case,

developers met with resistance by users and support groups alike who

continued to rely on their existing system. Developers did manage

to enlist user interest in small parts of the system, but it is not

known whether this eventually resulted (after the project was over)

in developing the support needed to sustain the system. In the procurement system case, it became clear that the evolving design would depend heavily on voluntary manpower from users and other institutions

in order to develop the technical materials users needed. Project

documents indicate that developers had difficulty enlisting this support, but a transfer strategy is not observable. Developers may have

needed to concentrate more on this factor since technical backup appeared to be such an important function in the design.

The SBA data base system was the only project where development

occurred primarily outside of the user-owner environment. A contractor was selected by the SBA central office and ETIP to conduct the

development. They designed two systems, one manual and one computerized, and essentially operated them during the entire project. As

in the other two ETIP cases, a transfer strategy is again not observable. Developers found that the manual system was unpopular and not

a feasible way to help solve user problems. It was deemphasized.

The computer system proved to be more successful with a small set

of users. However, it was expensive to operate and user support could
346

not be generated to continue its development or to adopt it into agencypractice. The central office also elected to end its support. Thus,

transfer may have been an unimportant issue as it became clear that

basic support for the system could not be obtained.
5.4 USER COMMITMENT CHARACTERISTICS (FACTORS 25-30)

5.4.1 Introduction

The final category of factors in the monitoring framework is

for measuring the extent of user commitment to the system. Commitment

to the system is viewed in this dissertation as a series of user decisions or actions over time which indicate increasing user interest,

support, and acceptance of the system. Full commitment to the system

will emerge when users elect to assume total responsibility for the

operational system and the development project is terminated.

This section identifies and describes six factors for developers to use in monitoring commitment. These have been divided into

two areas: factors relating to use of the system and factors relating

to support of the system. Monitoring use of the system involves examin

ing the type of applications made, the actions resulting from use,

and the extent of use across the organization. Prior to actual applica

tions during development stages, monitoring should be concerned with

user decisions in these areas and the progress made in designs. Problems in defining use may signal changing or uncertain user commitment

and require revisions in design or strategy. Later, as designs are

finalized and components become available for implementation, monitoring actual use provides further signs of increasing user commitment.

Wider, repeated applications indicate user acceptance, while disinterest, avoidance, or isolation of the system from routine organizational
348

business may indicate a decreasing commitment. Problems like these

in later stages may jeopardize reaching the ultimate goal of full

user operation.

Numerous other changes or actions in the user organization may

be needed during development to gain system acceptance and provide

for full operational support. Developers can monitor these supporting

actions for further signs of commitment. In early project stages,

the emergence of system champions who strongly support the development

and implementation of the system may be one of the first indications

of eventual commitment. Another indicator is the level of resources

users commit to the system. These allocations may first occur as

direct support of the project and include manpower and money. Later,

as components become available, users may decide to allocate resources

to operate the system. Developers should also monitor the user organization for changes that indicate support for system operations, such

as training programs or incentives to use the system, or changes made

as a result of system use, such as in decisionmaking styles or the

way the organization works. Changes like these indicate strong user

commitment to the system. The lack of essential supporting actions

or reactions may signal that commitment is minimal and full scale

operation doubtful.
5.4.2 System Use Factors

General Description

One theme in this dissertation is that user commitment to a system consists of a series of user actions over time which gradually

build to a full acceptance and support of the system. One set of

user actions indicating commitment relates to system use prior to

full implementation and the end of development. Early indicators

of use are the decisions made about the types and extent of system

applications users need. These influence both the design requirements

and the project control strategy. In later project stages, such as

testing and transfer, developers can monitor actual uses and gain

feedback on how well the system meets expectations. Problems at these

stages may have a significant impact on user commitment since they

are likely to be very visible. Developers need to closely monitor

use at this point so that the appropriate changes can be quickly made

in design or strategy.

Three factors are described in this section. First, developers

need to identify the types of applications of the system (or ones

it is expected to have) . This requires identifying system actions

with specific user tasks. The second indicator of use is the activity

resulting (or expected to result) from system applications. These

actions need to be documented as they will likely play a key role

in obtaining full user commitment to the system. Finally, developers
350

should monitor the extent of use in order to gauge how frequent and

widespread it is (or will be). This may be an especially important

factor to the top level managers who will decide whether to provide

the resources needed to fully support and institutionalize the system.

One ETIP system development case is used to illustrate these

factors. For simplicity, the case is described in one section after

all the factors are described.
5 . 4 . 2.1 Factor 25 — Applications

Description — The use of specific elements of the

system in specific units of work of the user organization.

Discussion

Developers need to identify the applications of a system to user

problems as one part of monitoring user commitment to the system.

In early project stages when designs are being formulated, developers

will have to rely on the expectations and plans about applications

rather than actual cases. These plans can provide important insight,

however, to the extent and type of commitment users expect to make.

Later, as system components become operational in testing and implementation stages, developers can then monitor whether the applications

are in fact occurring and are leading to full institutionalization

of the system. Problems or changes in applications may indicate a

need to revise the design or project strategy.

It is important that developers take a broad view of what constitutes an application. Many researchers confine their definition of

an application simply to the use of the system in a major decision

problem (Ein-Dor and Segev, 1978, p. 1065). This is a relatively

narrow view since it can not inadequately describe numerous supporting, underlying, or follow-on applications which may also be involved.

It is also possible that no major decision will occur (or does occur)
352

as a result of a system application. Thus, an indicator based on

use in a major decision would show no result when in reality numerous

applications have occurred to support the decision process (Ginzberg,

1978b, p. 60).

Developers should also include unexpected applications and areas

of non-use. For example, the availability of new data and capabilities may attract the attention of users outside of the primary user

environment. Modest additions or modifications of the system design

or project strategy may possibly solve some of their problems at a

low cost to the project. This could provide additional support to

the project and enhance the chances of obtaining full user commitment

to the system.

Similarly, avoidance of the system or non-use should also be

investigated. Users may perceive that the system will not (or actually doesn't) solve their problems. They may turn to other systems

and withdraw their support from the project (Comptroller General,

1978 ; Ruth, 1978 ). Developers may also find that staff who operate

the system are having difficulties doing so (Malvey, 1978, pp . 201-

220). They might prevent or seriously impair further operation in

an area unless the design is changed or they receive more training.

User resistance at these levels may have serious implications for

eventual user commitment to the system. Developers need to identify

problems with applications early and revise designs and strategy

accordingly.
353

The key problem in determining where applications occur (or will

occur) is to match some unit of work in the organization, such as

a decision, with some unit of system action. This matching can be

greatly facilitated by the use of models which detail the information

and decision flows of the user organization. Developers can take

these models and identify places where the system interacts with the

user. The range of applications can then be determined as can the

specific user behaviors which define work tasks.

It is advantageous to start modeling early, such as during initial problem definition stages when extensive contact with users occurs (see also Problem Identification in section 5. 3 -5.2). Developers

can then:

• Obtain an early indication of commitment by matching expected

system products and applications. This can be compared

to project goals (see also Boundary Factors, section 5.2.3).

• Plan the occurrence of applications as the project proceeds,

possibly starting with simpler ones and then moving into

more complex undertakings (see also Task Structure Factors,

section 5.3*2).

• Obtain a basis for establishing measures of actual use which

can be applied later during testing and implementation (see

also Project Control Factors, section 5.3.4).

• Uncover any different perspectives as to what constitutes

use. Differences among users may prevent agreement that

an application has occurred (or will occur) and have an effect on eventual institutionalization (see also Boundary

Factors, section 5.2.3).
354

5. 4. 2. 2 Factor 26 — Consequent Actions

Description — The actions resulting from areas of

direct system use.

Discussion

Further indicators of use are the user actions which come about

as a consequence of system applications (Young, Note 7, pp. 3-10).

These actions should be observable user behaviors which can be documented and shown to others. Changes in thinking may be a valuable

result of system use, but these are hard to measure and probably would

be considered weak evidence to justify support for a system. Credible

evidence is of critical importance to both users and developers.

User managers need proof that a system improves the operation of their

organization before they provide full support. Developers need to

evaluate performance and commitment and then determine whether changes

are needed in design or strategy.

There are several categories of actions developers might consider

monitoring in order to measure the effects of system use. These include the following:

• Follow on actions in organizational processes.

Developers can identify the linkages of various organizational actions and trace the impact of system use down the

line.

Organizational changes.
Use of the system may eventually bring about changes in

the organization, such as in structure or official procedures. Changes may occur in other systems or processes

which support system use (see Changes in the User Organization, section 5. 4. 3.3).

• Performance changes.

Factors which are used by the organization to measure performance might record changes which coincide with the introduction of a system. Performance improvements may consequently

lead to high level commitment to the system.

• Avoidance or non-use of the system.

Negative experiences in a particular application may lead some users to avoid further contact with the system, discontinue their support, or work against the system (see also

System Champions, section 5. 4. 3.1).

As with identifying system applications, determining where actions like these may occur and then measuring them can benefit from

initial detailed modeling of user processes. Models can help identify

the flow of information or decisions and provide developers with a

means of tracing user actions which emerge from specific system applications. Identifying these actions in advance also provides a means

of determining what decisionmakers consider valid evidence of use.

In addition, expected actions developed early through modeling can

be compared with actual behavior during use and provide a means of

measuring changes in user commitment.
356

5. 4. 2. 3 Factor 27 — Extent of Use

Description — The amount of use relative to the number

of potential applications.

Discussion

The third indicator for monitoring use is the extent of system

applications to user processes. Measuring extent includes determining

the frequency of use and the amount of use relative to the number

of potential applications. These factors should be monitored over

the life of the project and used to indicate changes in user commitment. Stable or decreasing levels of use might indicate dissatisfaction with the system design or loss of momentum in project strategy.

Revisions in both of these areas may be needed.

One major question that developers and users alike have to consider is how much use constitutes acceptance of the system. User

top management will likely have some threshold for the extent of use

above which they will give serious consideration to full support and

institutionalization. This threshold may be based on how often the

system is used over the range of different applications as well as

how this use affects the performance of the organization. Developers

can begin to uncover thresholds like this early by spending time detailing the types and extent of applications decisionmakers expect to

achieve. This may be best done by, again, modeling user processes
where the system is to operate and determining what use specifically

means. Developers should then continue to examine the extent of use

as designs are implemented and determine whether the expected levels

are achieved and prove to be significant enough to justify full institu

tionalization.

An ETIP Illustration

The ETIP-SBA project to develop a data base system illustrates

the three use factors discussed above. The goal of the project was

to provide a means for SBA field officials to increase the number

of government R&D contracts set-aside to small businesses. This involved developing a system to help match R&D contracts to small firms.

A contractor was hired to conduct the project and a system was built

in a short amount of time.

On the basis of the three use factors, the development approach

and the resulting design exhibited favorable conditions for obtaining

commitment to the system. First the application of the system was

fairly simple and well defined. The system provided procedures for

users in the field to classify both contracts and firms on similar

criteria. Matches on these criteria were defined and a score given

to indicate the overall degree of matching. The location of system

use within the set-aside justification process was identifiable and

standard in each field site. Thus it was easy for developers to identi
358

fy where applications would occur and how users would be involved.

It was also simple to promote use and to measure it.

Second, the results of use were also easily measurable. Setaside justifications based on use were to be submitted to agency contracting officials who then were to decide whether to make a set-aside

action. If they decided to set-aside part or all of the contract

to small businesses, well known, observable procedures were then used

to implement the action. Alternatively, if they decided against setting aside the contract, a series of appeals to higher management

levels were measurable. Thus the effects of system use were readily

observable. Increasing success in set-asides would be noticeable.

Finally, there were numerous potential applications for the system since set-aside procedures were similar across the many field

sites. Increasing commitment to the system could easily be measured

by the expanding number of uses at the different sites. In addition,

it was expected that repeated use of the system by field officials

would build upon itself and lead to a better system. Developers expected that the system would gradually obtain a data base of firms

that users could rely on for future R&D contracts. Thus commitment

would be further observable by the creation of individual data bases.

Developers found, however, that the system as first designed

was not attractive to users and that they could not induce user commitment to it. This was demonstrated during the field testing stage
when the strategy was to introduce the system to field officials and

to have it used experimentally in both retrospective and current cases

Testing was to have helped refine the system and build a track record

of use that would lead to increasing user interest. Unfortunately,

users considered the matching problem to be less significant than

the one of simply finding firms they could contact. In addition,

the matching process consumed a significant amount of the short time

available to make a set-aside justification.

More modest success was achieved later when the procedures were

computerized and a data base added from which to gather potential

firms. This change was the result of an opportunity in one agency

where officials needed to contract several projects quickly and felt

that set-asides were appropriate. The system proved very successful

in one retrospective test and was then applied, finally, to some real

cases which resulted in set-aside actions. The success of the applies

tions built considerable interest for the system in the one agency.

Unfortunately developers were not able to increase the extent

of use and build a wider commitment to the system. The system was

expensive to operate and direct users apparently did not have the

resources to support it on their own. The central SBA office could

not provide additional resources either.

In retrospect it would appear that the central office decision

to discontinue the project was in part based on the problems with
360

extent of use. First, developers had been unable to build a wide

consensus for the system in the field that would have probably been

attractive to central SBA decisionmakers. Second, R&D contracting

was only one of the many sectors that field officials had to review

for small business set-asides. Officials in the field and the central

office both felt that the computerized system was too costly for the

number of applicable cases.

This conclusion is further supported by the fact that the central

office elected to continue developing the system for a broader range

of applications. A computerized system, similar to the one for R&D

set-asides, was to be developed for all types of set-asides. Thus,

while the R&D system was unsuccessful, it did apparently demonstrate

the need for some system and caused some level of SBA commitment.

albeit in another form.
361

5.4.3 System Support Factors

General Description

Besides monitoring the direct use of the system to determine

user commitment, developers should also monitor other user actions

which indicate support for the system. These actions may include

early signs of user acceptance such as the emergence of system advocates as well as longer range institutional changes which are needed

to support the operation of a system. The lack of supporting actions

like these may indicate that the system is isolated from the user

organization and that it will not become fully implemented when the

development ends.

Three factors for monitoring system support are described in

this section. First, a system champion factor is presented. Champions are people who believe very strongly in the value of the system

and actively work for its development and acceptance. Their early

emergence may be one of the first signs of user commitment. A second

factor relates to the resources allocated by the user during development. Early signs of user commitment are indicated by resource allocations to project tasks while later signs are the support provided

to components made operational before the full system is completed.

A third factor concerns changes in the user organization which result

from system use during development or which are needed to support
its use. These include actions to establish training programs for

system operators as well as changes in how the organization works.
5. 4. 3.1 Factor 28 — System Champions

Description — The emergence of advocates for the system

in the user organization.

Discussion

An important source of support for a system development can be

a critically placed key man, advocate, or system champion (Keen, Note

2; Robey and Zeller, 1978; Thompson, 1978b, p. 332). A system champion is a person, perhaps best located in the user organization, who

believes very strongly in the system: its concepts, design, performance, etc. The system most likely matches the champion’s objectives

and the organization’s objectives. A champion has credibility inside

the organization, hopefully across different levels. His credibility

may even extend outside the organization as well. He can push the

system into existence and also find the pull for it from users (Thompson

1978b, p. 332). He should also be able to isolate the system from

attack by others (Libman, Note 8).

Developers should seek out system champions and cultivate their

participation. Their help can be a critical element for success through

out a project, especially in early stages when both the project and

the system are being defined. Key support from a champion at this

initial point may give the project the momentum and exposure it needs

to get started. System champions may offer the earliest sign of user
364

commitment and act as a catalyst for acquiring other user support.

Champions may emerge from different levels of the user organization and developers should actively seek this broad-based support.

Champions from top levels of management are clearly important to a

development since they may have great influence over resources, access

to the organization, or project strategy. It is especially advantageous to find a high level champion whose credibility extends both

up and down the lines of the organization. However, developers should

seek champions at other levels as well. This can help gain credibility for the system with key groups and also expose project teams

to different perspectives or expectations about system design or project strategy. These differences may present a barrier which developers should overcome. In addition, multiple champions, either within

or across organizational levels, may help reduce the potential disruptions caused by the loss of any one of them (Lucas and Plimpton, 1972)

(see also section 3-3.1.2 for an example case).

Developers must be skeptical, however, of champions who may be

overly supportive or controlling. They may be attempting to capture

the project, and thus the system, in order to use it for their own

personal rather than organizational objectives (Ackoff, I960, p. 262).

In cases like this, developers may find that the champion has little

if any credibility elsewhere and is regarded with suspicion from others.

Actions which are associated with the champion may not be well received

or desired elsewhere. Developers may ultimately find themselves locked
into a strategy which has very little chance of succeeding.

Developers must also be aware of those who actively oppose the

system. Just as the new system may have a key advocate, it may also

have someone who is uninterested or critical. Opponents may fight

development at each stage or stop it entirely. Alternatively, they

may carefully and slowly build up opposition to the system and bring

great pressures upon the project at key points, such as during decisionmaking on institutionalization. They may also elect to hamper an

operational system as much as possible or sabotage system processes

(Malvey, 1978, pp. 201-220). Developers should be open to the possibility that opponents exist, or will develop, and be prepared to actively counter the effects of their actions.

System opponents may not be limited to those who oppose development outright from the initial planning stages. Technical problems,

infighting already existing in the user organization, differences

of opinion over designs, or opposition to particular project decisions

may create new opponents during a development. In these situations,

it is critical that developers direct specific attention to the resolution of conflicts and consensus building. System developments can

involve change and disruption to users, and the creation of an active

opposition only makes strategy that much more complex and success

that much more difficult to achieve.
366

ETIP Examples

Perhaps the best example of a system champion is found in the

ETIP-PBS planning system development. The project was managed within

the special studies unit of the agency by the director, who was very

interested and active in the development. He believed that the concepts and the design of the system were central to agency business

and compatible with emerging agency policies. He was very active

in marketing the system to users, who were in line units outside of

the special studies group, and to high level policymakers, who had

to endorse and support the system. When user opposition appeared

for the system, he turned to outside groups who were interested in

concepts of the model and whose endorsement would have influence within his own agency.

As described in Chapter 2, however, this level of active support

was apparently not enough to gain full agency commitment to the system. Users continued to operate with their old system and agency

policymakers delayed approval. Difficult relations between the champion and user line units existed as well; the champion may have had

less influence than desired. It is possible that the project might

have benefited from additional champions in these other organizational

levels
367

5. 4. 3. 2 Factor 29 — Resource Commitments

Description — The type and extent of resources allocated by the user organization to support the system.

Discussion

A key indicator of user support for a system is the type and

extent of user resources committed during the development. The two

primary resources users can provide are manpower and money. In early

development stages, these might be supplied to the project to support

design and testing. Later, as subsystems become available for implementation, user resources may be supplied to operate them. Each of these

allocations is a sign of user commitment to implement a system after

the development is completed. Developers should not rely totally

on these signs, however. Many system projects have been characterized

by large commitments of user manpower and funds during development,

only to result in failure when the time came for full scale implementation and support.

There are several characteristics concerning user allocation

of money to a project which developers may want to monitor. These

are

• Temporary vs. permanent allocations.

In some cases, part or all of the development may be funded

out of special user project funds and be combined with re- sources originating from outside third parties. A change
368

in budgeting, where the system becomes a line item in a user budget, may be an important indicator of emerging support (Ein-Dor and Segev, 1978, p. 1071).

• Origin of money.

Developers may want to compare the origins of financial

resources with the location of primary users. Arrangements

not following organizational lines could mean broad support

or potential ownership problems. In addition, if multiple

sources are involved, developers may want to consider the

relative proportions of allocations versus expected use. Continuing support from all parties may be essential for

full scale system implementation.

• Planned allocations/expenditures.

Developers may want to examine how user allocations are planned over time and in what system areas they apply.

Targeted resources may signal important areas where user commitment will be based. User expenditures should also

be monitored. Success in the development will most likely

be followed by increasing resource allocations (Ein-Dor

and Segev, 1978, p. 1071).

These factors may not apply in projects which are entirely based on

third party funding and where user resources are expected only after

successful development.

Similar characteristics can also be monitored for user allocations of manpower. These may apply independently of any funding

allocations and be essential for strategies involving close user involvement. Developers should consider:

• Temporary versus permanent assignments.

User staff may only be temporarily assigned to project teams

or to the operation of initially available subsystems.

While these conditions would indicate an initial commitment,

a transition to permanent assignments to operate system

components would indicate a more significant level of support.
369

• Type of personnel assigned.

The development may require the skills and involvement of

users; and developers should monitor whether they are made

available. The assignment of key people in an organization

may signal a high degree of user commitment to the system

(see also Commitment, section 5. 3. 3.3).

• Level of involvement.

A key sign is also the extent of involvement. User personnel assigned to work on the system a small part of the time

may be too distracted by other business to contribute effectively to project objectives. Allocation of significant

portions of staff time may indicate a solid commitment to

the system. Commitment would also be indicated by increases

in the proportion of their time.

ETIP Examples

The ETIP projects had different strategies for user commitment

of resources. The PBS and SBA projects involved both agency and ETIP

funds for project initiation and it was expected that the agencies

would assume full support of the systems at the end of the projects.

These partial allocations did not result in final commitment to the

systems, however. In the SBA data base system, developers were unable

to obtain the support of field users near the end of the project and

the central office elected to withdraw support when project funds

ran out. In the PBS planning system, developers were unable to gain

support from line users, who continued to support their own system,

or from higher level managers, who could not support the system as

normal agency practice.

The NASPO-NIGP procurement information system project depended

entirely on ETIP funding throughout the development. It was expected
370

that as the system’s capabilities became available, users would begin

to provide resources and at project's end they would be fully supporting it. The gradual transition to user support did not occur, however, as developers had problems meeting user needs. It was also

discovered that direct users were not necessarily the only decisionmakers involved in allocating resources to system support. In some

cases, legislative officials had a key role in the budgeting of procurement agency operations and their support had not been considered in

the system development strategy. The project ended without being

able to obtain user funding to continue development.

The strategies for involving manpower also differed across the

ETIP projects. In the PBS projects, development was essentially conducted outside of the user environment and did not depend on gaining

user involvement in design or operation until the system was available. Users kept their distance from the project and continued operating their existing system. A similar strategy was used in the SBA

project until the field testing stage was reached. At this point,

developers planned to train users in the operation of the system and

have them apply it to both retrospective and current cases on their

own. User problems with the system prevented this kind of involvement

and commitment to the system did not evolve. Developers were forced

to operate the system and attempt to gain test cases on their own.

The lack of widespread support and involvement probably was a key

factor in the central office decision to discontinue its support for

the system.
371

In contrast, the NASPO-NIGP procurement information project depended on close user involvement in order to develop the system.

The strategy was to enlist volunteers from the pool of eventual users

to participate on task forces and develop prototype system products.

Since both NASPO and NIGP were national organizations representing

users, developers were able to obtain volunteers, many of whom were

recognized as leaders in procurement. Developers found , however

that the volunteer approach worked very slowly. Volunteers were available only for short time periods and the project competed with their

routine agency business. Also, the evolving difficulties in meeting

user needs under the project strategy probably made participation

less attractive. Several task forces could not produce the expected

products; in other cases where a package was developed, they were

not always attractive to users who had not been involved in their

creation. Developers learned, probably coo late, that more manpower

was needed which was fully dedicated to the project and which was

located in-house.
372

5.4.3 «3 Factor 30 - Changes in the User Organization

Description — The alteration of policies and procedures in the user organization in order to support

system operation.

Discussion

Other indicators of user support and commitment to a system are

the changes which take place around the system. Systems are placed

into complex organizations involving intertwined lines of communication, support, responsibility, and activities. Changing over to a

new system likely causes and/or requires changes in other places and

systems. New relationships and functions may be established. Developers should look beyond the immediate areas of system impact to other

areas where change may indicate a positive or negative force for user

commitment

There are numerous potential changes inside the organization

which developers can monitor. Some may occur in the areas which are

to support the operation of a system. For example, the system may

require new types of personnel or new positions which must be arranged

for by personnel divisions. Some users may need training in order

to operate the system and new programs may be created for this. Another

similar change is the creation of an incentive system either to attract users to the new system or to acknowledge improved performance

because of its use. Other changes may occur in processes, such as
decisionmaking style (McGrath, 1970) or how the organization works.

Changes may also be observed at higher management levels. For example, the system may support a new policy which high level decisionmakers must acknowledge and support. Finally, developers should examine changes in competing systems (Ackoff, I960; Thompson, 1978b,

p. 329). The continuation of competing systems may indicate a lack

of support for the new system, while conversely, the gradual withdrawal of support from them may indicate the opposite.

Developers should also monitor actions or conditions outside

of the user organization which may affect user commitment to the system. For example, outside institutions may mandate the use of the

system. This is especially the case in the public sector where new

systems can be supported by Congress or upper levels of the Executive

branch. Outside organizations may also have a role as system users

or suppliers of information. Changes in their roles or activities

may be a direct or indirect sign of system acceptance and support

in the user organization.

ETIP Examples

Perhaps the best example of organizational change among the ETIP

system developments is found in the PBS planning system project.

In this case, the system was developed around a policy that had not

as yet been fully implemented within the agency. In order to gain

credibility for the system, developers went outside the agency to
374

higher level policymakers responsible for promulgating new policies.

Developers hoped to demonstrate the system's compatibility with the

new policies. It was expected that their support would have a significant influence over the agency's subsequent commitment to the system.

Even though the compatibility was demonstrated, agency policymakers

reserved their support in order to fully examine the implications

of the new policy in their organization. Thus the system remained

isolated within the special studies unit in which it was developed.

Line units continued to operate their existing system. Little training occurred to educate line users in the use of the system and no

incentives were created for them to make the changeover. In addition,

it remained difficult to obtain data to support system functions since

the system required figures aggregated in a manner different from

normal agency practice. These requirements exceeded the capabilities

of existing agency data bases and there was little incentive to make

changes.
5.5 THE MONITORING FUNCTIONS

5 . 5.1 Introduction

The following sections describe the five functions the monitoring

framework can serve in a system development. As described earlier

in Chapter 4, background studies of ETIP projects and the literature

demonstrated that a monitoring process was needed by managers of complex developments to assess and revise strategy as the project unfolded. The framework was thus designed to serve two main administrative

functions, that of problem identification and strategy development.

The background studies also showed, however, that the monitoring

concept might also be useful for researchers interested in studying

system developments. In particular, monitoring could assist researchers in several ways

• Provide a means to study system developments in real-time

rather than retrospectively.

• Obtain better empirical evidence on developments that has

been called for in the literature.

• Provide a means to test and utilize literature based con- cepts in actual projects.

• Promote the use of research and research processes by project managers.

These ideas suggested that the framework might also serve in a research

function at the same time it was serving project managers.
376

With the addition of two more supporting functions, documentation

and dissemination, the resulting monitoring framework was thus designed

to serve five different functions. Each of these functions is described

below in more detail. In addition, a final section presents a general

model of how the framework, via the functions, might be implemented

within a specific project.

5.5.2 Function ifl — Problem Identification

Problem identification is the major function for the framework,

reflecting the ETIP management need for a procedure to anticipate

and pinpoint key problem areas. The framework is structured to contain key elements of a systems project, areas where problems usually

occur and which are important to eventual project success. By routinely monitoring these areas, managers can identify problems more

quickly and possibly earlier than would be the case without a framework. Use of the factors may also promote easier recognition of problems. In addition, problems which are new, complex, or multi-faceted,

may be more easily decomposed into recognizable, manageable components

by analyzing them with the framework factors.

5 . 5.3 Function #2 — Strategy Development

The second major function for the framework is strategy development. The performance of a strategy can be monitored through the

elements of the framework. Problems and progress identified in these
377

areas may highlight the need to change some or all of a strategy in

order to continue progress towards objectives. If strategy problems

occur, the factor structure and the information base available from

regular monitoring may then also assist in formulating the changes

needed. In addition, the framework can promote better strategy by

offering a means to consider longer term pespectives. First, retrospective analysis of the factors may uncover problems that are unidentified by constant attention to immediate circumstances. Similarly,

by focusing on specific framework factors during strategy revision,

managers may be encouraged to predict the effects of strategy changes

in the future. Besides bringing a longer term perspective to problem

solving, this can provide an opportunity to identify the conditions

or points in time when a strategy should again be reviewed. The longer term cause and effect linkages which are identified through these

analysis may also help managers avoid repeating mistakes.

5.5.4 Function #3 — Research

The third function of the framework is to provide a means for

researchers to study and contribute to on-going system developments.

This reflects the finding that systems researchers see a need for

closer contact with actual projects — partly to gather empirical

evidence not easily available through other means and partly to transfer the guidance available in the literature to practitioners who

some claim have difficulty using the literature. While these two

problems are to some extent exacerbated by the few incentives practi-
378

tioners have to study their own experiences, it does appear that an

approach which unites researchers and practitioners may be a good

way to gain the access needed by the research community.

The monitoring framework provides a research opportunity by establishing a data collection process that researchers can use to define

and implement studies. By adapting their studies to the framework

structure and procedures, researchers can acquire a series of data

points at the same time as project managers. They can then use this

data to study changes and linkages between factors of interest and

to produce valuable insights of use to others.

Researchers may find it beneficial to design their studies with

the project managers. Besides helping to ensure continuing access

to the project, this approach may provide researchers an opportunity

to help managers with problem solving. Research studies could be

designed to support problem solving. In addition, researchers might

be able to bring in relevant guidance from the literature to help

managers. This would help counter the claim that research in the

systems literature cannot be used.

5.5.5 Function #4 — Documentation

The fourth function of the framework is to document project activities for the administrative and research purposes discussed above.

Documentation here means establishing an organized and stable written
record of project activities as they occur in the key selected areas

of the framework. In the short term, this function can help project

managers identify current problems and progress and assist in finding

ways to improve the project. For researchers, documentation of project events as they occur is one essential feature of the monitoring

approach that makes it attractive over retrospective studies.

Documentation is seen as particularly useful, however, for tracing and analyzing selected factors over long periods of time. For

managers, documentation of project activities can be inefficient for

short term problem solving. In addition, documentation over the long

term can provide several benefits that may be difficult to obtain

from short term analyses of immediate actions:

• Routine documentation over the long term can help managers

identify special, evolving, or recurring problems.

• Longer term analyses of management actions may help identify

successful and unsuccessful approaches, promoting improved

strategies or new ideas for current actions.

• Long term documentation can support decisionmaking in the

project, particularly for major decisions at the end of

phases. These decisions may rely on credible evidence of

performance — unattainable from quick retrospective analyse

at the time of the decision.

• Documentation can be used to inform new staff or project

history and thus ease the problem of turnover.

For researchers, the long term stream of data available from

documentation provides the data base on which to conduct research.

In particular, long range documentation provides :
380

• The evidence upon which to identify and test relationships

between important factors in development.

• A basis of developing models of the dynamic, evolving process of a large scale system development presently missing

in the literature.

• Empirical evidence about all project stages that is also

needed in the literature according to some researchers.

This may be especially important for other researchers who

need better access to actual projects.

5.5.6 Function #5 — Dissemination

The fifth and final function of the monitoring framework is to

promote dissemination of information about the system development

to groups not directly involved in oversight roles. The framework

structure and resulting data base can facilitate the transfer of information to these groups by making it easier and quicker for managers

to generate the information needed. This of course must be tempered

with the need to avoid release of interim or uncertain information

which might harm the project.

Managers may find the dissemination function useful both inside

and outside a project. For insiders, dissemination of monitoring

information can facilitate a common awareness of problems and progress

and help promote united actions. In a large project, this may be

especially important in coordinating groups working on different parts

of a system. Dissemination of pertinent information about progress

may also have the secondary effect of promoting or facilitating the

monitoring activity. Project staff may come to rely on periodic re-
ports of activities for guiding their own work.

For outsiders, dissemination of monitoring information can be

essential to keeping groups informed about progress and building awareness and support for the system. This may be especially critical

to user groups not directly involved in the development, but having

some indirect role in future system operations. Other important groups

can be top level managers or outside institutions who supply resources

to the project, periodically review progress, and approve continuations of the work. These people may need current and retrospective

reports on development activities, both easily supplied if monitoring

has been routinized.

Dissemination is also important to researchers. By providing

the opportunity to closely study on-going development activities,

the framework can facilitate the transfer of information to the systems literature. For example, the framework can help provide a more

dynamic, evolving view of development and other empirical evidence

that some researchers believe are needed in this area. Besides helping other researchers, real time analysis of project activities may

offer insights to other system developers who are looking for guidance

of use in their own situations.
382

5.5.7 A Model for Applying the Framework

The framework has been designed with a set of procedures that

managers can use to apply it to specific developments. This process

is modeled in Figure 5.3 by a flow diagram of events which are generally expected to be a part of monitoring. The model is illustrative;

modifications would likely be needed in order to match the process

to the project.

As shown in Figure 5.3, the monitoring process begins when project managers identify the need for monitoring and proceed to specify

the indicators, procedures, and staff needed to operate the monitoring

system. To specify the monitoring indicators, managers need to examine the individual elements of the framework and decide what indicators are needed. For example, under the category of user commitment,

managers need to identify events that will indicate use and support

for the system. For the system champion factor listed under support,

this will involve identifying where champions would arise and what

actions will be considered supportive. This identification process

will of course be influenced by what managers and others, inside or

outside the project, need to know during a development. In addition,

managers may find gaps in the elements or alternative factors which

2

better match their specific situation. Analysis of each framework

2

As mentioned earlier (section 4. 2. 6.1), the framework is intended

to be comprehensive. However, this dissertation is still the first

attempt at defining such a framework. Revisions and additions are ex- pected .
FIGURE 5.3 A GENERAL MODEL FOR APPLYING THE MONITORING FRAMENORK
384

element in this manner will specify where monitoring should occur

and what needs to be monitored.

Next, managers need to specify the procedures for monitoring.

Some of the considerations include the following:

• How monitoring will be coordinated with major decision points

in the project.

• How monitoring data will be collected.

• Who will receive monitoring information and what information

will be supplied.

• The storage and location of monitoring information.

• What kind of documentation is desired.

• How research activities will be tied to management activities .

Operational details in these areas will establish how monitoring is

to be performed and integrated into project management.

Finally, at the same time as these procedures are being explored,

managers should also identify the team which will perform the monitoring. While top level project managers are the main users of monitoring information, their role in collecting the information should be

minimal. This work should be delegated to the staff supporting project

leaders. Managers will have to decide who will be on the team and

how the different groups in a development should be represented.

It will be beneficial if the team consists of members from all participating groups: the user organization, the development team, and
the group of researchers (if present) studying the development. This

should help ensure collection and use of monitoring information, while

improving the credibility of the monitoring by bringing different

perspectives to the process.

After these initial preparations are completed, the monitoring

process is started. Problems and progress in the specific areas of

framework are identified as they occur or at selected periodic reviews.

If the situation is new or not well understood, the monitoring team

may need to decompose project events into smaller elements by identifying the relevant factors in each framework category. The relationships between different factors should also be considered. As appropriate, the monitoring team then produces a problem statement for

project managers. This procedure is illustrated in Figure 5.4.

With the problem identified, managers then decide what, if anything, should be done. This is when the strategy development function

of the framework can be useful. If managers know what should be changed,

then no further analysis is needed and the changes should be made.

In this case, the only remaining tasks for the monitoring team are

to: document the problems and actions taken for future reference,

identify future conditions when these actions should again be reviewed,

and disseminate information as needed about the management activity.

3 It is assumed here that situations occur which require intervention

by project managers.
386

Identify Initial

Event of Interest

Problem Definition Using the Framework

Problem Components by Category and Relationships

Design Process Commitment

Relevant Factor 1 t

Relevant Factor 2

\

unexpected

relationship

Relevant Factor 3 Relevant Factor 4

Unexpected Factor 5 t

I no relationship

Irrelevant Factor 6

^

Produce Problem Definition

For Managers

Based on Factors

FIGURE 5.4 PROBLEM SOLVING USING THE FRAMEWORK
However, if the appropriate action is uncertain, further analysis

of the problem using the framework may be beneficial for identifying

options. In this case, the framework factors may provide a focus

both for the problem and the management intervention. Managers can:

• Identify the key components of the problem using the elements of the framework and what actions might be targeted

at these areas.

• Identify and examine the relationship between problem elements, using the different framework categories to decompose

a complex situation.

• Examine guidance on the problem elements from previous monitoring information, project research underway, or other

outside sources.

Once the interventions are specifically identified, the monitoring

team then determines how to trace the effects in the future and whether

special attention is needed to identify when a review of the changes

may be needed (see illustration in Figure 5.5). Over the long run,

tying the state of the project to management interventions should

help the monitoring team identify successful approaches and promote

better management control.

After the changes are made and plans for future monitoring are

established, the monitoring team documents the activities and disseminates information about them as needed to others inside and outside

the project.

This pattern of monitoring continues during the project until

monitoring is no longer needed by managers. At this time, the monitor-
Ijncerlain

I

n

te

rveiil

Ion

388

FIGURE

5.5

STRATEGY

FORMULATION

USING

THE

FRAMEWORK
ing team completes documentation of the project, emphasizing the final

state of the project in terms of the framework elements. Researchers,

if present, then conduct analyses of the data collected by the monitoring team according to their earlier plans. The monitoring team should

also review the framework to determine how well it worked and whether

changes are needed for future applications.

Finally, the results of the project are disseminated as appropriate to other interested researchers and practitioners.

5.6 CONCLUSION

This chapter has presented a framework that can be used to monitor important areas of a system development. Thirty factors have been

identified for the framework along with five functions monitoring

can serve in project management.

The framework factors have been divided into three main categories — design, process, and user commitment characteristics. Design

characteristics are factors which refer to what a system looks like

and how it operates. Developers need to monitor the design since

it can greatly affect whether users accept and implement the completed

system. Process characteristics are factors which refer to the procedures, resources, structures, and personnel used to conduct and control the development. System developers need to monitor these characteristics because they also have an effect on user commitment to the
system. In addition, the development process must be tailored to

the evolving requirements of the design. Finally, user commitment

contains behavioral measures of system use and support in the owner

organization. These are actions which can occur throughout a development and are intertwined with the design and process characteristics.

Developers need to monitor commitment in order to learn how it changes

during a project and whether it is building up to full acceptance

of the completed system.

The framework functions have been designed to support both adminis

trative and research objectives in a system development. The five

functions include problem identification, strategy development, research, documentation, and dissemination. Each function has been

described separately. A model of how the monitoring process might

be implemented in a project has also been presented. This model has

shown how the monitoring functions would work and how they would relate to each other.

The following chapter presents a test of the framework in a differ

ent ETIP case involving the development of a large evaluation system.

Chapter 6 presents a brief history of the project and an illustration

of how the framework might have been used by managers during one selected period.
391

CHAPTER 6

AN APPLICATION OF THE FRAMEWORK

TO A SYSTEM DEVELOPMENT

6.1 INTRODUCTION

This chapter presents the application of the monitoring framework

to a specific ETIP system development. The purpose of the application

is to illustrate the usefulness of the framework in the management of

a large and complex system project. This is shown by first mapping the

framework onto a selected period of the project where one particular

problem tended to dominate events. The mapping is then used to show

how the framework might have helped managers more effectively identify

and resolve the problem. A more complete description of the applica

tion design is presented in Chapter 4.

The chapter is divided into two parts. First, a general chronology

of the ETIP project is presented to familiarize the reader with the case.

The project involved the three phase, multi-yeared development of an

evaluation system for ETIP procurement experiments. While the project

lasted over five years, only the first three were aimed at development

of the system. This period is presented in the most detail in the

chronology.

The second part of the chapter describes how the framework might

have been used during one period of the project to improve project management. The first six month period of the project was selected as the
392

example because it contained a significant problem, that of establishing a conmunication system among participants, which dominated events

and affected many areas of the project. The application shows how

the framework might have contributed to an earlier and more effective

resolution of the problem. The analysis includes descriptions of

the evolution of the problem, specific components of the problem,

effects on other areas of the project, how the problem was actually

resolved, and finally how the framework might have helped managers.

Readers will note t^iat only one period out of the three main

years of the project has been selected for illustrating the framework.

This example focuses primarily on the process factors of the framework

and to a much lesser extent involves a few of the design and user

commitment factors. Other examples were expected to illustrate design

and commitment more completely. However, time limits on the research

prevented further selection of examples.

6.2 A BRIEF CHRONOLOGY OF THE EVALUATION SYSTEM DEVELOPMENT

0.2.1. Introduction

This part of Chapter 6 presents a brief chronology of the evaluation system development. The purpose of this review is to familiarize the reader with the basic events of the project.

The project history is divided into four sections. First the background and objectives are reviewed. Then each phase of the project is

described. A table is provided on the next page which briefly outlines
Date Event

September 1975 Chief, Experimental Methods, joins ETIP

and begins planning evaluations of experiments .

February 1976 Project plans for the development

of an evaluation system for procurement

experiments approved. RFP announced in

Commerce Business Daily. (Two separate

contracts, agency impact and commercial

impact evaluation).

June 1976 Two contracts signed (agency impact - Contractor A, commercial impact-Contractor C).

July 1976 Phase 1 work begins - initial meetings.

October 1976 Contractors submit first deliverables - plans for Phase 1.

March 1977 Phase 2 planning begins. Reports expected

for Phase 1 identified.

April 1977 Contractors prepare preliminary outline

of prototype evaluation system.

June 1977 Contractors submit combined prototype design

and management plan for Phase 2. COTR

approves Phase 2 for both contractors.

October 1977 Phase 2 begins. Prototype system begins

operation.

January 1978 Local Washington, D.C. office established

for the system. System manager appointed.

June 1978 Contractors submit Phase 3 plans to ETIP.

July 1978 System designed to answer a set of 16 questions.

September 1978 Phase 3 begins. Phase 2 extended to April

1979.

FIGUPE 6.1 BRIEF CHRONOLOGY OF THE ETIP PROJECT TO DEVELOP

AN EVALUATION SYSTEM FOR THE PROCUREMENT PROGRAM
394

January 1979 Draft system design issued for comments.

February 1979 System pretested with several potential

users.

March 1979 System design completed for Phase 2. Phase

2 ends.

May 1979 COTR leaves, new COTR appointed.

June 1979 Contractors submit Phase 3 plans, propose

quarterly reviews to update system. Phase

3 extended to April 1980.

March 1980 COTR leaves. New COTR appointed.

September 1980 Phase 3 contract revised - update of evaluation system in Phase 3 no longer required.

December 1980 Contracts extended into 1981 to complete

several technical reports.

FIGURE 6.1 (Continued)
395

•"e ^o

f(7

•C-. V/.

’ -"s,

/<> 's

ij. ''^o°3

-O

>>£,'/% j. ';,°-3 d- 0 d~ ^o V

V.^*%

^e, /v

V

) V

^5/

>0. ^0' % ‘'tf.

‘'0, ‘O^'^f

/

' <7

d

'*vrX:

c*

'''

V “*9

'o

• S X-

(j

^0/

X*5'

<3%

% '’d

‘'^,

Oj. *^0, ,X

- S©

"-*> r a\

FIGURE

6.2

MAJOR

EVENTS

AND

PIL\SES

OF

THE

EVALUATION

SYSTEM

DEVELOPMENT

ONA

TIME

LINE
396

the major events of the project. A figure is also provided to present

these events graphically.

Readers should note that names of individuals have not been used

in the chronology. Only institutional names are provided, and in the

case of the contractors, artificial names are used. In addition, citations to project documents in order to verify events have not been used.

Instead, the basic set of events on which the summary is based is included in the appendix.
6.2.2 Background and Objectives of the Project 1

6. 2. 2.1 The Emergence of an Evaluation Program in ETIP

The impetus to develop an evaluation system began several years

before the project was initiated. Early in 197^, ETIP managers became

interested in more explicit integration of evaluation into ETIP projects. Earnest planning of evaluation activities increased later

in September 1975 with the arrival of a methodological specialist

2

to head a new evaluation area in ETIP called Experimental methods.

During the ensuing months, the Director of ETIP decided that

money should be made available to contract for evaluations of projects.

This decision was partly based on the obvious problem that ETIP did

not have the staff to conduct evaluations itself. While the COTR

proceeded to plan an evaluation program with this approach, he also

felt that a capability for conducting evaluation should be developed

within the government rather than continually contracting for this

support. This led to the thinking that the impending evaluation projects be conducted to both develop this capability and transfer it

to agencies involved with ETIP work. This approach was later transformed into the idea of developing an evaluation system which could

be institutionalized.

Portions of the description in this section are adapted from a more detailed chronology developed by Libman (1980, pp. 68-149).

2 This person later became the ETIP manager in charge of the evaluation

system project. The official project manager's title was to become con- tracting officer's technical representative (COTR). Hereafter, COTR

will be used to identify this person.
398

6. 2. 2. 2 Evaluation in the ETIP Procurement Program

The first application of the evaluation system approach occurred

in the Procurement Program area of ETIP. The Procurement Program

was conducting numerous projects with a variety of Federal, state,

and local government agencies. These projects were designed to test

different procurement incentives in these agencies and determine their

effectiveness in both obtaining better products for government use

and stimulating the comnercial sector to improve their products.

For example, many projects were designed around the addition of life

cycle costing to an agency procurement (such as to buy air conditioners for agency use). In this case, ETIP and the procurement division

of the agency would replace the simple initial cost criteria used

to evaluate industry bids with a cost criteria that reflected both

initial and on-going costs for operation or maintenance. Incentives

like this were expected to affect the kinds of products proposed by

industry, such as to encourage the offering or development of more

energy efficient products in cases where energy consumption cost was

now added to the procurement process.

Developing an evaluation program and system for the procurement

area was not a straightforward project however. The requirements

for evaluation were broad, changing, and complex. For example:

• The Procurement Program was highly fluid, in that the number

and type of experiments (and agency partners) were variable.

• There were numerous objectives for evaluation from specific

and immediate (e.g., what happened in an experiment) to

broad and long term (e.g., the implications of a series

of experiments for policy changes in the agency).
• There were numerous stakeholders in the proourement work,

including Federal, state, and local partner agencies, and

related public and private institutions with a stake in

procurement policies.

These characteristics contributed a high degree of uncertainty

as to what evaluation would be needed as well as the eventual design

of a system to conduct evaluations over the long term. It was also

unclear how many systems would be needed and who would be the owners.

While ETIP had an interest in the long term results generated by evalu

tions from the system, it was not to be the primary user or owner

of the system.

6. 2. 2. 3 Developing a Work Statement

The high degree of uncertainty associated with the type and extent of evaluation to be conducted as well as the system design resulted in the development of a flexible work statement for the prospec

tive project. By the spring of 1976, a Request for Proposals (RFP)

for acquiring contractor support had been developed using a flexible

statement. The work statement emphasized seven kinds of activity

3 ETIP expected to occur, such as management or data collection, but

did not include specific design (either for evaluations or the system)

requirements. However, an illustrative set of items which could be

produced in the contract was included with the work statement to help

describe the needed work.

3 The others included reports and reviews, objectives, background,

evaluation design, and data analysis.
400

The prospective procurement evaluation work was divided into

two main areas (and two contracts): one for conducting evaluation

of agency impacts and one for evaluation of commercial impacts. Agency

impact evaluation was to be concerned with the immediate and subsequent effects of ETIP procurement experiments or other related changes

on the agency partners and others (Thompson, 1976, p. 1). Commercial

impact evaluation was to be concerned with the immediate and subsequent effects of ETIP procurement experiments on the commercial sector

and possibly on other related markets (Thompson, 1976, p. 1).

The general set of objectives was the same for each project

and reflected the need for specific evaluation support as well as

for development of system. These objectives included (Thompson,

1976, p. 1):

1. An overall description of the experiments as well as selected

detailed descriptions.

2. An overall assessment of both the immediate and subsequent

effects (impacts) of experiments, as well as selected detailed assessments.

3. An evaluation system or process which can be used by the

appropriate government agency (ies) to obtain evaluations

on a continuing basis of these as well as future, similar

experiments

The projects were divided into three phases, with the first two

of each funded at a total of approximately one million dollars.

Phase 1, to last 15 months, was generally to focus on the evaluation

of selected experiments and the design and pilot testing of the related evaluation system (Thompson, 1976, pp. 2-3). Phase 2 was to
continue specific evaluations, refinement of the system, and testing

of a prototype system. Phase 3 was to focus on a "turn key" implementation of the system(s) by the appropriate agency and was to be funded

when requirements were more certain. The phasing of the projects was

provided as a formal checkpoint to determine whether the money was

being spent effectively.

6. 2. 2. 4 Awarding the Contracts

With the RFP completed, a process was then implemented to procure

contract support. This process was known as parametric factor evaluation (PFE) and had been adapted for the procurements by Thompson (1976).

PFE involved establishing a panel of evaluators to assess proposals

on a selected set of factors. These factors had been determined earlier

during development of the RFP after extensive interviews with ETIP

staff and reviews of program documents (the seven factors mentioned

above) . A number of proposals were received and a bidder was selected

4

for each contract.

Contracts were awarded on June 30, 1976, and work was scheduled

to begin immediately in July, 1976.

Readers interested in the PFE process used in these procurements should consult Libman (1980).
402

6.2.3 Phase 1

6. 2. 3.1 The First Six Months of Phase 1, July-December 1976

The first several months of the project were focused on various

start-up activities. During July and August, a number of meetings

were held to introduce the people involved with the project to each

other (including ETIP staff, partner agency staff, the new contractors, and other contractors). These meetings were also used to exchange background information on ETIP activities and to begin planning

the activities of Phase 1. Special concern was placed on project

reporting requirements, schedules, identifying who (on the contractor

teams) was responsible for different tasks, and the process ETIP expected to use for monitoring and reviewing progress.

Project activity broadened considerably following these meetings

as contact between ETIP, the contractors, and the PAAs increased.

Various requests for contractor support began to reach the COTE from

the Procurement Program of ETIP and the PAAs as they identified their

priority evaluation needs. Several of the requests were selected

to begin work on specific technical projects. The contractors also

devoted time to the preparation of the first quarter reports which

were to specify their ideas and plans for Phase 1.

In late September, the contractors submitted their first reports

to ETIP. Since these were the preliminary results from the contrac-
403

tors, ETIP and the PAAs reviewed them closely. Generally, the reaction to the reports was mixed. The management plan from Contractor A

was considered very weak and it appeared to demonstrate little or

no gains in knowledge of ETIP projects. In contrast, the management

plan from Contractor C was much more detailed. However, it left out

information on how key decisions had been reached. It was also apparent from the overlap of the reports that the contractors were not

adequately coordinating their efforts.

These problems prompted a series of meetings over the next several months as ETIP attempted to obtain improvements in the reports

and to educate the contractors on ETIP needs. While some progress

was made toward these ends, other problems also began to surface.

For example:

• The contractors were trying to pursue second quarter tasks

while the first quarter results were still under question

(the COTR had not yet accepted the reports).

• ETIP continued to press for more detailed financial information that the contractors were not obligated to produce.

• Some PAAs were giving mixed reviews of contractor interviews,

emphasizing a seeming lack of knowledge about the purpose

of the project and the payoffs for PAA participation.

• The ETIP Director began to comment on what he believed to

be the poor performance of the contractors.

These circumstances added considerable pressure to the continuing

reviews of contractor reports and progress, and communication became

more difficult. Finally, in mid-November, ETIP and Contractor C met

face-to-face to identify and resolve the range of problems. At the
meetings, Contractor C emphasized the difficulties they were having

determining ETIP objectives in the project, specifying the results

expected by ETIP, and meeting the growing paperwork requirements.

ETIP commented on is frustrations over the responsiveness of the contractor.

As a result of the meeting, ETIP and Contractor C agreed to several changes in project structure and communication procedures that

would reduce paperwork and return attention to the central technical

work of the project. Contractor C then developed a new management

plan to reflect these changes. Contractor teams were reorganized

and new milestones were proposed.

Similar changes were instituted by Contractor A over the next

5 several months.

6. 2. 3. 2 The Second Six Months of Phase 1, January- June 1977

The second six month period of Phase 1 was more focused on the

specific technical efforts of the project as attention increased on

the contractor support requests made by ETIP and the PAAs. One specific project, an evaluation of an experiment with Federal Supply Service (PSS), was highlighted at this time since it appeared to be an

important Phase 1 product and required complimentary efforts by both

5 The communication problems discussed here really dominated the

project during the first six months. This time period was thus used

as a test case for the monitoring framework (see section 6.3).
405

contractors. ETIP and both contractors began to work jointly on the

conduct of this evaluation.

By the end of March, the COTR was considering the design of the

approval process for Phase 2 initiation since Phase 2 could be started

while Phase 1 was still underway. With the range of activities now

occurring, the COTR had to determine which Phase 1 contractor products

would be essential for approval and what schedule of completion was

needed. A preliminary list of products and dates was then developed

and distributed to each contractor. After several refinements, these

lists were accepted by the contractors as the final requirements for

Phase 2 approval, then scheduled for June.

Attention in the April to June period became increasingly focused

on these expected products. Two of the main reports the COTR wanted

were the prototype system design developed in Phase 1 and preliminary

management plans for Phase 2. In early April, the COTR decided that

each report should be a joint effort from the contractors since Phase 2

activities (i.e., implementation of the prototype evaluation system)

would require joint efforts by the contractors. The COTR thus requested the development of a joint design and plan from the contractors

and organized several workshops to facilitate coordination.

By mid-May, the first full drafts of these two reports were submitted by the contractors. Several changes for Phase 2 were proposed.

First, the contractors proposed that the management of the project
406

in Phase 2 be accomplished by a steering committee composed of ETIP

and contractor staff. This structure was aimed at improving the coordination of the various groups as well as providing a regular forum

for joint discussion and decisionmaking. The COTR was to be the

chairman and final decisionmaker on the committee. This idea was

accepted by the COTR and led to use of the steering committee approach

during the rest of Phase 1.

A second proposal from the contractors was to restructure the

evaluation system design into a set of procedures. Based on the types

of evaluation activities required in Phase 1 (such as evaluability

assessment), these procedures were to form the subsystem components.

This idea was also accepted by the COTR and work began reformulating

the prototype design to be reported in Phase 1.

As work on the Phase 1 reports continued in May, it became apparent to the COTR that the workings of the Phase 2 management plan

were not well tied to the evolving design of the system and its Phase 2

development. The COTR therefore decided to have the plan and the

design reports combined into one document which would then be used

as the primary vehicle for management control in Phase 2.

By mid-June, the contractors submitted the combined plan-design

report along with other technical reports on specific Phase 1 projects.

While the COTR was not completely satisfied with the main Phase 2

plan, he felt that the Phase 2 approval requirements had been met
407

and he thus approved Phase 2 for both contractors. Phase 2 was expected to begin around September.

Several changes were made by the COTR for Phase 2 on the basis

of contractor performance in Phase 1. Most significantly, the COTR

reduced the Contractor A funding level by 2056 from earlier plans and

transferred the amount to the Contractor C effort. This reflected

the COTR's desire to shift more of the system design responsibility

to Contractor C in Phase 2. Contractor C was thus given the role

of system manager in the continued development of the system.

6. 2. 5.

The End of Phase 1, July-September 1977

The last three months of Phase 1 were mostly concerned with the

transition from Phase 1 to Phase 2. With the preliminary structure

and plans for Phase 2 activities set in June, ETIP and the contractors

began to draft Phase 2 work statements which would be signed by the

contractors

During this work, further refinements in the prototype system

design were made. It was decided to divide the evaluation system

into two main components — a routine system and a non-routine system.

The routine system was to meet the standard, well known, and cn-going

information needs in ETIP experiments. The non-routine system was

to handle new types of information needs that required special skills

and novel evaluation design work.
408

Division of the system into these two components then had to

be coordinated with the division of responsibilities and resources

set by the COTR in June. Since Contractor C was to become the overall

system manager in Phase 2 and the routine system was the main component of the prototype system, the COTR assigned further development

responsibilities for the routine system to Contractor C. The COTR

then divided the responsibilities of further non-routine system development to both contractors according to their respective agency and

commercial impact domains. Contractor C thus received the bulk of

responsibilities and resources for Phase 2 while Contractor A was

primarily limited to activities concerning the non-routine system.

The Phase 2 work statements were then accepted by the contractors

and operation of the prototype evaluation system under Phase 2 resources

began in September. An interim system manager was appointed by Contractor C with the understanding that a permanent system manager would

be hired and located in the Washington, D. C., area within the next

several months. It was also decided by the system management committee that the project would be aimed at a go/no-go decision on institutionalization of the system within PAAs in Phase 3 somewhere around

February, 1978.

Phase 1 work was mostly completed by the contractors at the end

of September. However, in order to complete final reports summarizing

Phase 1 activities and lessons, both contractors were given no-cost

extensions of their Phase 1 contracts.
6.2.4 Phase 2

6. 2. 4.1 The First Six Months of Phase 2, September 1977-March 1978

Phase 2 began with the formation of several new project teams

as part of the management structure. Based on proposals made in

Phase 1, these teams included one for each PAA experimental program,

one for institutionalization of the system, and one to examine ETIP’

role as a future system user. These teams began to meet during the

final months of 1977 and assumed control over most of the technical

projects underway.

The formation of the institutionalization teams also marked an

increased focus on institutionalization in the project. As decided

earlier, ETIP expected to make a go/no-go decision on a permanent

Icoation for the system somewhere around February, 1978. The team

thus began to formulate objectives, strategies, and milestones in

relation to this goal. It was agreed that an important need in further project work was to focus system activities more on the key decisionmakers in partner agencies who might support the system in the

long term. Several actions were then taken in this direction in

December

• A new information request procedure for system action was designed with the requirement that decisionmakers needs

had to be clearly specified.
410

• The management committee directed the system manager to

develop on evaluation subsystem to follow-up on system

products

• The PAA experimental teams increased their focus on decisionmaker needs and related requirements in subsystem designs.

• The COTR began to interview one PAA to determine how the

system might become more integrated with agency decision

processes in the long term.

The COTR used information from these various actions to begin drafting

an institutionalization plan.

This was followed in January by a review of Phase 2 plans and

status in order update the direction of the project with the evolving

institutionalization work. A particular concern at ETIP at this point

was the need to establish a local office for the system in the

Washington, D. C. , area so that closer contact could be made with prospective system owners. ETIP reemphasized the need for a local system

manager to Contractor C. A local manager was then proposed and established in February along with the office. Besides taking control

of the routine system, the manager also began interviewing agency

decisionmakers to help develop a system requirements document.

The COTR also became concerned around this time over the amount

of project work being devoted to the design of new experiments or

the identification of new agency partners. While this work was important, the COTR felt that it was limiting the development and testing

of other evaluation procedures on already established experiments.

The OOTR thus decided in February to establish two accounts for the
411

project. One was devoted to experiment design or "front-end” work

and the other to further development of the evaluation procedures.

In addition, the information request process used to access these

resources was tightened by requiring a detailed (but informal) work

statement and budget with each request. Allocation of project resources was thus more significantly controlled such that the system

continued to be developed while some support was also offered on new

experimental starts.^

6. 2. 4. 2 The Second Six Months of Phase 2, April-September 1978

The next several months of the project involved further refinement of the system design in a manner which mostly set the agenda

for the rest of Phase 2. In April, the COTR issued the first draft

of a plan for Phase 3 in which he outlined a set of questions he expected the system to be able to answer (i.e., a list of information

needs for evaluation). This was followed by a series of design reviews where ETIP and the contractors examined technical activities

and subsystems.

As these reviews continued into May, it came to be decided that

the questions proposed by the COTR would be the primary design requirements against which the evolving system procedures would be measured.

r° The need for new starts was a priority for the ETIP Procurement

Program as many of the experiments in Phase 1 were close to completion .
412

Specifically, it was agreed that:

• The set of questions would be further explored and that

a refined list would be used to test the system in Phase 2.

• The number of questions would eventually be fixed for the

test, while others would be added afterwards if needed.

• The subsystems then under development would constitute the

system for Phase 2.

• The system response to further information requests in

Phase 2 would be limited to products similar to those

already developed.

Thus, to a great degree, the system design was fixed for the remainder

of Phase 2 and the specifications for a test were detailed.

With this structure established, the system manager then began

to interview PAA staff and to analyze Phase 1 and 2 reports to further

refine the set of questions for the system. By the end of June, the

system manager and the COTR refined the set into 14 basic questions.

Plans for testing the system on these questions were then discussed

for Phases 2 and 3. With Phase 3 approved within ETIP by this time,

the COTR also requested Phase 3 proposals from the contractors.

In July, a new information request procedure was designed around

the set of basic questions. Several new requests from ETIP and the

PAAs were tried in July and August as a preliminary test of the procedure and the rest of the system. It became clear in September, however, that Phase 2 would have to be extended (it was scheduled to

end at this point) in order to more fully develop the system proce-
413

dures and conduct the desired tests with PAAs. The COTR thus extended

Phase 2 to March 1979.

Phase 3 was also initiated in September as acceptable plans had

been received from both contractors. The COTR then requested Phase 3

management plans.

6. 2. 4. 3 The Last Six Months of Phase 2, October 1978-March 1979

As work on system procedures and questions continued during the

October to December period, ETIP and the contractors also began discussions on the final reports for Phase 2 and plans for Phase 3.

It was decided in December that three main reports (besides the other

specific technical reports related to individual experiments) were

needed. These included a system design report prepared jointly by

the contractors, a Phase 2 final report from Contractor A, and a combined Phase 1 and 2 summary report from Contractor C. It was also

decided that, where technical work on specific evaluations was expected to continue past Phase 2, some projects would then be transferred

to Phase 3 accounts.

As work then proceeded on these reports and the completion of

most other technical activities, the system manager began the system

test which was the final major task for Phase 2. The test consisted

of obtaining user reactions to the potential usefulness of the system

design in their procurement programs. The system manager met with
414

several high level officials of agencies who might use the system and

described parts of the design. These potential owners were then asked

whether the system would meet information needs they had in the conduct of their procurement programs. Generally, the system manager

found that:

• Potential users (or system owners) considered the basic

system questions as important, continuing information needs

in their procurement programs.

• The procedures outlined in the system seemed logical for

obtaining answers to these questions.

• The data sources for obtaining information seemed valid.

These results concluded testing in Phase 2.

Phase 2 was finally completed in March with the submission of

the reports planned earlier. All of these were accepted by the COTR.

The system manager and the COTR then planned a review of the Phase 3

objectives starting in April.

6.2.5 The Remainder of Phase 3, April-December 1979

The remainder of Phase 3 was marked by a number of changes in

ETIP management and goals, which had an effect on the directions of

the project. By May, the COTR had resigned from ETIP and been replaced

by the Chief of the Procurement Program (who had been involved with

the project from the beginning). The new COTR began working with

both contractors to define studies for the rest of Phase 3*
By mid-June the system manager, coordinating both Contractor

A and C, submitted plans for Phase 3 work. The system manager also

proposed that quarterly meetings be held to update the system with

procedures from on-going technical work. These plans were accepted

by both ETIP and Contractor A. The new COTR considered himself the

primary turnkey user of the system in Phase 3 and moved quickly to

7 use the system to generate new experiments.

The search for experiments continued until October, when several

candidates were selected by the COTR. The system manager then began

to look for additional staff to assign to these new areas. Both contracts were extended to April 1980 so that these new projects could

be finished. Generally, little if any system development work was

conducted during this period.

The project changed again in March 1980 when the COTR resigned

and another COTR was appointed. By this time, the Procurement Program

was itself coming to a close. Some technical work on procurement

projects continued in the following months to complete work already

started.

The turnkey declaration of the new COTR actually represented a significant change for the project. Originally, the goal of the pro- ject had been to institutionalize the system in one or more of the

ETIP partner agencies participating in procurement experiments. The shift to ETIP at this point essentially reduced further efforts toward this goal.
416

In September 1980, the new COTR assigned several studies to the

contractors which were unrelated to the procurement work conducted

earlier. A contract change was then made to eliminate an update of

the system as a requirement for completing Phase 3* Eventually, continuing work on the new studies required further extension of both

contracts into 1981.

6.2.6 Conclusion

This section of Chapter 6 has presented a brief review of the

ETIP project to develop an evaluation system. First, the background

and objectives of the project were reviewed. Each of the three phases

of the project were then discussed.

The purpose of the chronology has been to familiarize the reader

with the basic events of the project. This general understanding

is useful in the next half of Chapter 6 where one narrower period

of time in the development is used to illustrate the monitoring framework.

6.3 APPLICATION OF THE MONITORING FRAMEWORK TO ONE SELECTED PERIOD

OF THE SYSTEM DEVELOPMENT

6.3.1 Introduction

This section describes the application of the framework to one

period selected from the evaluation system project. The purpose of

the application is to demonstrate how the framework could have been
417

useful to project managers during this period. In the retrospective

approach to the application, usefulness is shown by mapping the framework onto project events and suggesting ways that the framework might

have contributed to the resolution of critical problems.

The period selected is the first six months of the project where

managers had problems establishing an effective system of reports

and reviews for the project. This problem tended to dominate the

project and had effects on many other areas of the project. The dominance of the problems with reports and reviews and the complex relationships to other areas of the project appeared to be an excellent situation in which the contributions of the framework could be demonstrated.

The application is divided into several sections. These include

the following:

• An initial summary of the major points in the application.

• A background description on the evolution of the problem.

• A review of the specific components of the problem.

• An analysis of the relationship between the problem and

other areas of the project (these other areas are identified

by framework factors).

• An analysis of how use of the framework might have helped

with the problem.

Readers may find it useful to first review the summary and concluding sections of the application before scanning the details of the

intermediate sections.
6.3.2 Reports and Reviews, July-December 1976

6. 3. 2.1 Introduction and Summary

A major area of concern during the first six months of the project was the establishment of a system of reports and reviews for project communications. Early emphasis was placed on developing the

system in order to assure adequate communication between the various

groups involved (i.e., ETIP, contractors, PAAs, and other groups)

and to identify the specific reports expected during Phase 1 of the

project. ETIP especially wanted a system of communication that would

ensure close interaction with the contractors over the numerous technical uncertainties involved with developing evaluations and the evalu

ation system.

The initial system for reports and reviews developed during the

first quarter led to many communication problems, however. These

included too much paper work, misunderstandings over the types of

reports required, and disorganized feedback concerning progress.

In addition, the first set of major reports submitted at the end of

the first quarter was initially considered poor by ETIP staff.

Activities and problems with the reports and reviews system also

affected other areas of the project. In terms of the monitoring frame

work, these areas included:

• Design characteristics — system boundaries.
419

• Process characteristics — task size, task priorities, team

personnel commitment, organization and responsibilities,

and decision points and milestones.

• User commitment characteristics — system champions.

Use of the monitoring framework during this period might have

helped establish earlier a more workable system for reports and reviews. A project monitor located in ETIP might have been able to

identify the growing problems with the initial system and its increasing effects on other areas of the project. Alternative approaches,

possibly similar to those eventually established in the second quarter, might then have been suggested. In any case, the time and energy

expended in creating and revising the first approach to reports and

reviews might have been alternatively channeled into work on the central technical problems of the project and accelerated progress overall.

6. 3. 2. 2 Establishing the System for Reports and Reviews

Expectations concerning a system for reports and reviews were

established early by ETIP in the RFP for the project. In this document, ETIP proposed that a reporting system be developed by the contractors within the first quarter (Thompson, 1976, p. 3^). Parts

of the proposal included the types of reports expected (such as for

progress or system design) and a rough schedule for when these reports

might be produced.
An important part of the reports and reviews system was also

to be a process of progressive review and approval (Thompson, 1976,

p. 49 ). ETIP wanted to have a close involvement in project activities

(COTR, Note 1; ETIP, Note 2), particularly because of the technical

uncertainties involved with developing evaluations and an evaluation

system. ETIP proposed to minimize the significance of reports in

the evolution of the technical work and instead emphasize more continuous interaction over time. Final reports were mostly to document

already established and approved progress.

These expectations were reemphasized during the first several

meetings with the contractors (ETIP, Note 3j Note 4). In addition,

ETIP began to propose other ideas for the system which the contractors

were to consider. These included:

• A specially structured financial accounting system.

ETIP wanted a financial accounting system that would enable

project managers to control resources as well as learn the

costs of performing various evaluation activities (COTR,

Note 5). This was seen as essential to developing the re- quirements for the eventual evaluation system.

• The use of report dummies.

As part of its desire for close involvement, ETIP proposed

that a dummy mock-up of each expected report be established

early and updated as it evolved (ETIP, Note 6, Note 7; Contractor C, Note 8). These dummies would be located at both

ETIP and contractor sites. Final reports would thus be

based on the final state of the progressively refined dummy.

• Using reports to market the project.

Reports were expected to be used both for internal coramunica

tion and external marketing of the project (system) to poten

tial system owners (Thompson, 1976, pp. 46-49; COTR, Note 9)
ETIP proposed to make reports available to everyone unless

there were confidentiality problems (ETIP, Note 10).

0 • Using the RFP factor structure for reporting.

ETIP proposed using the seven factor structure of the work

statement as the overall structure for the types of reports

expected (ETIP, Note 11, Note 12, Note 13). The illustrative work statement was also used to discuss the specific

reports needed. Finally, the COTR emphasized ETIP’s desire

to use the factors to measure project performance.

Many of these proposals were later adopted by the contractors

as they developed their management plans for Phase 1. Both contractors endorsed the ideas of progressive review and close interaction v;ith

ETIP and felt they were essential to successful conduct of the project

(Contractor A, Note 14, pp. 51-52; Contractor C, Note 15, p. II-6).

Report dummies for the first quarter reports were developed and the

factor structure was accepted to broadly define the kinds of reports

to be produced (Contractor A, Note 16; Contractor C, Note 17). Progress and first quarter reports followed the factor structure closely

(Contractor C, Note 18, Note 19; Contractor A, Note 20, Note 21).

6. 3. 2. 3 Problems with the Reports and Reviews System

Despite the early attention to reports and reviews, the resulting

system began to have serious problems at the end of the first quarter.

As discussed earlier in the chronology (section 6.2), seven factors

(management, reports and reviews, background, objectives, data collection, data analysis, and evaluation design) were used to define

the statement of work in the RFP. These factors specified the types

of activity to occur in the project, not specific evaluations or system designs. An illustrative work statement was also included that

proposed more specific tasks.
422

The primary event at this time was ETIP dissatisfaction with the first

quarter reports from both contractors (COTR, Note 22, Note 1). Contractor A had not provided much detail on their plans for Phase 1 nor

had they demonstrated much of a gain in knowledge of ETIP activities

since the beginning of the project. Contractor C provided more details and plans for their effort but did not include supporting information on how they had selected priorities. In addition, several

reports which had been expected were not submitted (e.g., preliminary

evaluation designs from Contractor A or industry background analyses

from Contractor C).

Over the next several months, ETIP and the contractors attempted

to resolve these problems through a series of reviews and written

feedback. While some understandings were reached during this period,

the difficulties with the reports or with the ensuing discussions

highlighted several problems concerning the reports and reviews system. In particular, ETIP and the contractors found that there was:

• Too much paper work.

The series of monthly, quarterly, special, contact, and

financial reports were becoming very burdensome on the contractors, but ETIP was also frustrated because it couldn’t

obtain the information it wanted (ETIP, Note 23, Note 24).

This problem may have been enhanced in part by the large

physical distance between the groups, resulting in fewer

face-to-face meetings.

• Misunderstandings over the reports required.

Both contractors had assumed that the items in the illustrative work statement were required (COTR, Note 25;
423

Contractor A, Note 16 ; Contractor C, Note 17; ETIP, Note

23). While this had not been the intention of ETIP (ETIP,

Note 6), the contractors felt that numerous signals from

ETIP reinforced their assumption.

• Delayed commitment to or acceptance of reports.

Contrary to the spirit of progressive approval and acceptance, ETIP was delaying acceptance of first quarter reports

while work continued well into the second quarter. In addition, the contractors felt it was difficult to obtain commitment to report dummies or outlines as final versions were prepared (ETIP, Note 26; Tape of Meeting, Note 27).

• Disorganized feedback on reports.

The contractors felt that feedback was very slow from the

various groups in the project (i.e., ETIP divisions and

PAAs) and that it was not easy for them to process it without better coordination (ETIP, Note 23; Tape of Meeting,

Note 27). It was also unclear how to treat feedback from

the COTR (e.g., as a requirement or a consideration) (COTR,

Note 28), or who had the final authority to make changes.

In short, the system for reports and reviews were exhibiting

many of the characteristics which ETIP and the contractors had attempted

to avoid during .ne design of the system.

6. 3. 2.Effects on other Areas of the Project

Problems and actions concerning the system of reports and reviews

also had effects on other areas of the project. For example, contractor adoption of ETIP proposals for reports and schedules (in the illustrative work statement) essentially set the team structure and tasks

for Phase 1 before the contractors were able to formulate their own

proposals. In another example, the numerous paper work requirements

established by ETIP and the contractors eventually led to noticeable
424

frustrations in the project teams who were attempting to produce these

documents

The following sections describe seven areas of the project which

were primarily affected by events with the reports and reviews system.

These areas correspond to elements listed in the monitoring framework,

and include the following:

• Design characteristics — system boundaries.

• Process characteristics — task size, task priorities, team

personnel commitment, organization and responsibilities,

and decision points and milestones.

• User commitment characteristics — system champions.

The relationship of these areas events concerning reports and reviews

are summarized in Figure 6.3.

Boundaries . The definition of boundaries for the evaluation

system was affected to some extent by the first quarter actions related to reports and reviews. As described below in Task Priorities ,

the COTR in Experimental Methods requested that the contractors develop a report on the priorities for evaluation tasks in Phase 1.

In order to facilitate the development of the report, the COTR attempted to minimize or delay the initiation of work on specific experiments. In addition, the COTR emphasized that he was primarily interested in the identification of evaluation needs in on-going experiments rather than the needs for experiments still under design at

that time (COTR, Note 29).
ESIGN

CHARACTERISTICS

PROCESS

CHARACTERISTICS

USER

COMMITMENT

425

FIGURE

6.3

RELATIONSHIP

OF

ACTIVITIES

CONCERNING

REPORTS

AND

REVIEWS

TO

OTHER

PROJECT

AREAS
426

These requests were not completely supported by the Procurement

Program or the PAAs. During this initial period of the evaluation

project, their support needs related primarily to the selection and

design of new experiments and they were very interested in having

the contractors provide this support (Transcript of Meeting, Mote

30; ETIP, Note 10; Contractor C, Note 3D* While these needs were

not entirely outside the scope of the project and it was expected

that the evaluation system would eventually provide this type of support, the COTR and Experimental Methods staff felt that the involvement with new experiments would divert too much attention from overall

planning and the identification of existing evaluation needs in experiments already underway.

The controversy over the COTR approach produced some uncertainty

in the Procurement Program and PAA staff over whether the system would

ever support experiment design (Contractor C, Note 32, Note 33; ETIP,

Note 3^). This uncertainty was further enhanced by the reluctance

of the contractors to become involved with the design of new experiments, an activity they considered outside of the evaluation system

(ETIP, Note 35; COTR, Note 36).

Task Size . While both contractors technically were free to examine the needs of ETIP and the PAAs and to determine the tasks for

Phase 1 (COTR, Note 22; ETIP, Note 37), decisions concerning reports

and reviews early in the project broadly defined the tasks before

their examinations were completed. As discussed above in section
6. 3 *2. 2, ETIP proposed in the first several weeks that the seven factors used in the RFP work statement (management, reports and reviews,

background, objectives, data collection, data analysis, and evaluation

design) be used to structure the tasks of the project for Phase 1.

The items in the illustrative work statement, specifying relevant

activities and reports for each factor, were also included in the

discussions. As a result, both contractors quickly adopted this structure and defined their tasks in Phase 1 according to the illustrative

items (Contractor A, Note 1&; Contractor C, Note 25, Note 17). Since

many of the illustrative items described reports to be produced at

certain times, this early decision to adopt the illustrative work

statement increased attention on deliverables much before analyses

had been made on which experiments and evaluations would be supported

by the project. The emphasis in both ETIP and the contractors was

more on meeting the schedule of deliverables than on the underlying

activities they required.

Task Priorities . The adoption of the illustrative work statement

items described above essentially set the general task priorities

for Phase 1 and led to a schedule of reports to be produced by the

contractors. One of these reports in the first quarter was to outline

the specific priorities for evaluation in Phase 1 (i.e., which experiments were to be evaluated) and provide the supporting justifications

for selecting these priorities. The Experimental Methods staff were

especially interested in this report because it was to provide essential information on the experiments not available within ETIP at the
428

time (ETIP, Note 38; COTR, Note 21, Note 22). The COTR thus attempted

to minimize or delay the initiation of work on specific experiments

until this report was completed.

However, as reported in Boundaries above, this approach caused

some concern in the Procurement Program area and the PAAs because

it prevented attention to their immediate needs for support. During

this period, these two groups were attempting to design several new

experiments or obtain key information on activities of existing experiments (e.g., a survey of industry reaction to a proposed experiment).

They felt that the emphasis on the priority report, especially one

aimed at identifying evaluation needs of existing experiments, was

overlooking their primary interests. Eventually, to help reduce this

controversy, the COTR decided to allow some work on experiment design

if it was simple and inexpensive (ETIP, Note 10).

Despite this early concern over the report on priorities, the

reports submitted by each contractor several months later were very

unsatisfactory to the Experimental Methods staff. Contractor C’s

report described six specific projects for priority attention but

did not include the detailed supporting justifications expected by

ETIP. These choices were also a surprise to the PAA involved in the

projects (ETIP, Note 38). Contractor A's report did not provide a

specific list of priorities or the justifications. This unsatisfactory outcome led the COTR to request extensive revisions in the reports before they would be accepted.
Dissatisfaction with the reports also raised some problems for

the system of reports and reviews. ETIP had expected to be closely

involved with the technical work of the project. In this particular

case, close involvement meant being able to observe the project review

process implemented by each contractor in the determination of priorities. The lack of details on the process in the reports or during

their preparation frustrated the COTR (COTR, Note 39). In addition

to this problem, the request for extensive revisions in the reports

before acceptance countered the expectation that final reports would

primarily serve to document already established progress, thus avoiding the problems of revisions after the fact. The lack of acceptance

added further frustrations to both the contractors and ETIP as some

attention to first quarter issues had to be continued during the second quarter.

Team Personnel Commitment . Problems concerning the system of

reports and reviews began to have an effect on the morale of the contractor teams in the second quarter. Contractor A reported in December

that the extensive documentation requested by ETIP was frustrating

the professionals on project teams (ETIP, Note 24). These professionals felt that the dummy approach to reporting was a waste of time

since it required submitting documents that would soon be improved

anyway (ETIP, Note 42). Later in December, the project leader for

Contractor A reported his own frustrations over txhe continuing delays

of their teams in producing reports for ETIP (ETIP, Note 26). These

delays were partly attributed to unsatisfactory communications with
430

ETIP, especially in obtaining decisions from the COTR. The project

leader added, however, that their internal frustrations were not a

sign of decreasing interest in the project; on the contrary, the leader emphasized their growing enthusiasm for the project.

Similar concerns were also evident with Contractor C. At a meeting with ETIP in November over ways to improve communication with

ETIP, the senior level supervisor from Contractor C commented on how

the project used to be fun, but was becoming a chore because of all

the paperwork (Tape of Meeting, Note 27).

Organization and Responsibilities . The organization of contractor teams was directly affected by events concerning reports and reviews. As discussed above, both contractors adopted the factor structure of the work statement in organizing their tasks, and this also

led to a similar structuring of their teams. Within the first several

weeks of the project, team leaders were assigned to each factor area

and reports relevant to the area (as proposed in the illustrative

work statement) were identified. Only Contractor A reported that

it felt that staff responsibilities did not fit neatly within the

factor structure and that some other approach was needed (ETIP, Note

1 ).

The division of responsibilities between ETIP and the contractors

was also affected by events in reports and reviews. The contractors

perceived a dual role for the COTR: one as the ETIP decisionmaker
431

specifying requirements they had to meet and another as an evaluation

stakeholder trying to help solve the technical problems of the project. The contractors had difficulty deciding which role they were

seeing in the feedback from the COTR and felt that they had to respond

to every comment (COTR, Note 28; Contractor C, Note 40). They also

felt that the extensive and repeated feedback from the COTR conflicted

with the often stated COTR approach of wanting the contractors to

run their own projects (i.e., conduct they work they were expected

to perform) . The problem eventually was solved by having the COTR

identify which role he was taking in all future feedback on contractor

progress and reports (COTR, Note 4l).

This confusion over feedback from the COTR reflected a much broader problem: that of a subtle shifting of decisionmaking responsibility from the contractors to ETIP. This was enhanced by several other

ETIP actions concerning reports and reviews:

• Focusing attention early and regularly in the first quarter

on deliverables.

• Emphasizing the deliverables proposed in the illustrative

work statement.

• Emphasizing the RFP factor structure for reporting while

at the same time emphasizing that the contractors should

design a reporting system that made them comfortable (ETIP,

Note 13).

• Proposing detailed financial reporting requirements which

later led the contractors to feel that ETIP was trying to

second guess them (COTR, Note 42).

Identification of these actions and their effects on the division

of responsibilities eventually led to a restructuring of the project
432

by ETIP and the contractors in the second quarter.

Decision Points and Milestones . As mentioned several times above,

both contractors adopted the factor structure and the respective proposals on reporting that were included in the illustrative work statement. This structure also included a quarterly schedule for reporting

in Phase 1 in which major reports for each factor were expected in

three month intervals. These quarterly dates for reports and related

activities thus became the milestones for the project. ETIP also

planned major reviews of progress at the end of the first two quarters .

This schedule of activities and reports was later questioned

in the second quarter when elements of the first quarter report were

found to be unsatisfactory by ETIP. Subsequent discussions of project

structure eventually reduced the emphasis on the quarterly milestones

in favor of more appropriate individual milestones for each specific

technical activity.

System Champions . Liaison with the partner agencies involved

with experiments was an important objective for the evaluation project

(Thompson, 1976, p. 47). Besides their importance to the design and

implementation of specific evaluations, some or all of the partner

agencies were expected to become eventual owners of the evaluation

system being developed by ETIP and the contractors. ETIP thus wanted

partner agencies closely involved in the system for reports and re-
views as a means of ensuring that evaluations were useful to the respec

tive agency and that they cooperated in the project.

However, the idea of using reports and reviews to help market

the project (and eventually the system) to important agency staff

was not developed in the first six months. Contractor C recognized

the importance of PAAs to the project (ETIP, Note 43) and proposed

that reports be used in marketing (COTR, Note 44), but later decided

that they key reports of the first quarter would be aimed at ETIP,

which would then decide how they should be distributed (Contractor C,

Note 45, p. 8). Contractor A appeared reluctant to have the PAAs

involved in reviewing deliverables even though the COTR emphasized

to them that PAA cooperation was essential to the project (COTR,

Note 9)

As a result, the feedback process on the first quarter reports

was disorganized. For example, one major PAA (the Federal Supply

Service) found Contractor A's first quarter report useful but long,

and was uncertain how to provide feedback to the contractors (ETIP,

Note 46). For Contractor C’s report, the same PAA was surprised by

the six projects selected for priority attention and had to suggest

alternatives after the fact (ETIP, Note 38). These problems did not

demonstrate a well targeted process for reports and reviews that was

promoting PAA cooperation. They also led to frustrations on all sides

over the inability of the reporting system to provide an orderly communication process among the numerous participants in the project (Tape
of meeting, Note 27).

6. 3 . 2. 5 Resolving the Problems with Reports and Reviews

During the last two months of the second quarter, ETIP and the

contractors revised the system of reports and reviews to improve communications and eliminate the problems arising from the initial approach

of the first quarter. These changes also affected the structure of

the project (tasks and teams) for the contractors for the remainder

of Phase 1.

Perhaps the most significant change involved the elimination

of the items and structure of the illustrative work statement adopted

by the contractors during the first several weeks of the project.

ETIP had not intended these proposals to be requirements the contractors had to meet, although the numerous references to them at meetings

and in official communications led the contractors to assume the opposite. ETIP reaffirmed that they were not requirements during the

last two months of the second quarter, and the matter was immediately

referred to the contracting officers of each organization to confirm

that this was the case.

Another major change at this time was that the COTR finally accepted the first quarter reports from the contractors as submitted.

The revisions desired by ETIP were now to be incorporated into future
reports as appropriate. This allowed both ETIP and the contractors

to focus full attention on the activities of the second quarter as

they were evolving.

Several other ideas were accepted by both sides to improve communications and reduce paper work. These included:

• Elimination of the quarterly schedule of reports in favor

of individual schedules for each specific technical activity.

• Elimination of the continuous "dummy" approach to progressive review in favor of a more finite set of working papers

and drafts and increased interaction.

• Use of monthly contractor reports as the primary means of

documenting progress.

• Encouragement of increased communication both formal and

informal. Staff in ETIP and the contractors who could make

commitments were now designated.

• Refinement of COTR feedback. The COTR was now to define

feedback in terms of requirements or suggestions.

• Use of the management plan as the primary device for controlling the project.

Following these agreements, Contractor C issued a revised management plan to firmly document the changes and establish a different

structure for their operations (Contractor C, Note 47). Contractor

C teams were now organized around specific technical areas or management, eliminating the factor structure of teams used in the first

quarter. As a result, task structure was also simplified: the broad

areas of the factor structure were now replaced by a simple list of
work areas and respective accounts. Monthly reports were also redesigned to follow this structure.

Similar changes were made for Contractor A and were reflected

in their management plan submitted later in the third quarter.

6. 3. 2. 6 How the Monitoring Framework Could Have Helped

The system of reports and reviews was a significant source of

attention and problems during the first six months of the project.

In the earliest weeks, ETIP placed great emphasis on the establishment

of a reporting system and the identification of deliverables which

were expected from the contractors. Given the physical distances

between ETIP, the contractors, and the partner agencies, the numerous

activities and people involved, and the ambitious objectives and technical uncertainties concerning evaluation and system design, a workable system of reports and reviews was clearly a priority need which

justified significant attention in the early months of the project.

However, the concentration on developing the reports and reviews

system did not produce an effective process. By the end of the first

three months, the resulting system was proving to be unsatisfactory

to both ETIP and the contractors. ETIP was not able to obtain the

information it wanted on contractor activities nor was it able to

effectively communicate its concerns to the contractors. On the other

hand, the contractors were becoming increasingly frustrated with the
excessive paper work requirements of the system and the extensive

and disorganized feedback from ETIP and PAAs which caused uncertainties on what was required in the project. Review sessions which attempted to resolve these problems were becoming more confrontative

and increasing the distance between ETIP, the contractors, and the

PAAs.

In addition, activities and problems concerning reports and reviews in the first two quarters were affecting other areas of the

project and broadening the impact on the project in general. Problems

were raised over the division of responsibilities between ETIP and

the contractors, cooperation with agency partners became endangered,

and uncertainties were raised over the general design of the evaluation system being developed. Eventually, these problems had to be

discussed extensively by project leaders in the second quarter in

order to reduce their effects on the conduct of work in general in

the project. A revised system of reporting was established along

with changes in the approaches to managing the project.

Given this history, the primary benefit of using the monitoring

framework in project management might have been to help establish

a more workable system of reports and reviews earlier in the project,

thus avoiding the cumbersome problems of the initial approach. A

project monitor located in ETIP might have been able to identify the

existence and growing impact of problems with reports and reviews

that project leaders may have been too busy to appreciate. Alterna-
438

tive strategies for the system, possibly similar to those eventually

developed, might then have been suggested much sooner.

For example, a project monitor might have;

• Monitored Team Personnel Commitment , thus identifying that

the excessive paper work requirements were promoting frustrations on both sides.

• Monitored Decision Points and Milestones , thus determining

that control and information transfer were not being accomplished with the quarterly reporting format.

• Monitored Reports and Reviews , thus identifying that ETIP

itself was not following its desires for the system of re- ports and reviews, and leading to contractor uncertainties

over the needs.

• Monitored Organization and Responsibilities , thus identifying the subtle shifting of control to ETIP enhanced by

events with reports and reviews.

• Monitored Task Priorities , thus determining that the contractors were not establishing the information base or involving

the people required for making decisions on priorities.

• Monitored System Champions , thus improving the ability of

the system of reports and reviews to enlist cooperation

of key agency staff and market the evaluation system.

Earlier monitoring of one or more of these areas might have helped

project leaders identify the growing ineffectiveness of the reports

and reviews system. In addition, disassembly, via the framework,

of the system’s effects into these other areas might have helped develop more effective approaches to the increasingly Intertwined problems of the system and the rest of the project. In any case, the

time and energy expended in creating and revising the initial approaches to reports and reviews might have alternatively been focused more
439

on the central technical problems of the project and approaches

to meeting them.

6.4 CONCLUSION

This chapter has presented a preliminary exploration of the usefulness of the framework in a specific ETIP development. The ETIP project,

an evaluation system development, was first reviewed to help familiarize

readers with events of the case. One selected period of the project,

the first six months, was then discussed using the framework.

The application demonstrated two qualities of the proposed framework. First, by mapping the framework onto events of the selected

period, it was shown that the factors were relevant to the case.

Second, using this mapping, it was then shown that use of the framework as a whole might have helped project managers identify earlier

a critical problem of the period and thus possibly have promoted more

efficient actions relevant to the problem. The application did not

show, however, the usefulness of the other functions of the framework,

including strategy development, research, documentation, and dissemination .
440

CHAPTER 7

CONCLUSIONS AND RECOMMENDATIONS

FOR FURTHER RESEARCH

7.1 AN OVERVIEW OF RESULTS

This dissertation has been concerned with the problems of managing

complex system developments where there are substantial uncertainties

in system requirements, development processes, and ultimate ownership.

The specific environment for the research has been the Experimental

Technology Incentives Program (ETIP) in which complex system developments have been conducted as part of its research program with other

government agencies.

It was shown within the ETIP environment that managers needed

a mechanism to monitor the numerous and evolving activities that a

complex development contained. The monitoring process specifically

had to help managers:

• Sense or anticipate problems earlier among the numerous,

interrelated events of a development.

• Help tie together short and long range perspectives when

solving specific problems.

• Establish an information base for the formulation of more effective management control interventions.

It was further shown that the extensive literature reviewed on

systems development did not contain an explicit model of monitoring

that could be readily adapted into the ETIP environment. Moreover,
441

few specific approaches to the monitoring problem in general were

uncovered. Instead it was found that the literature was separated

into several distinct areas relating to system development — that

of case studies of developments, development strategy models, important factors in developments, and system designs arising from specific

situations. Each area emphasized a different perspective on developing a system and concentrated on selected sets of factors which were

important considerations for developers. For example:

• Few sources recognized the broad range of factors potentially important in a development (and thus important for monitoring) .

• The changing and evolving circumstances of a complex development were rarely integrated with discussions of the important factors developers should consider (thus defining when

factors should be monitored).

• Little attention was given to the process by which users come to accept and support a system and how this process

is intertwined with development activities (and thus important for monitoring also).

While some sources in these four different areas recognized the need

for a monitoring process, these discussions were not yet integrated

with the specific knowledge already available. Several sources in

the literature confirmed aspects of this problem by calling for more

research on managing complex developments and the development of models managers might use to control the complex circumstances of their

own situations.
442

A monitoring framework for ETIP has thus been proposed both as

a mechanism for ETIP managers and as an initial attempt to respond

to the management research needs of the literature. The framework

contains a broad range of factors to be monitored and is accompanied

by a set of procedures for these factors. The procedures are divided

into five related functions which reflect the administrative and research purposes monitoring can serve. These functions include:

• Problem identification — The tracking and assessment of

key areas where development problems typically occur, both

in the short and long range.

• Strategy development — The identification and development

of explicit management actions to solve problems and revise

strategy

• Research — The design and implementation of real-time studies of the development process.

• Documentation — The establishment of an organized and stable recording process which can identify special or recur- ring problems, and support decisionmaking and research.

• Dissemination — The distribution of key development information to those inside and outside a development as a means

of facilitating coordinated actions and distributing knowledge gained.

Thirty factors important for monitoring a development are identified for the framework based on studies of ETIP projects and the systems literature (see Figure 7.1). The factors are divided into three

main categories:

• Design characteristics — Factors related to the design

and functioning of the system.

• Process characteristics — Factors related to the methods,

organization, personnel, resources, and procedures used

in the development process.
443

to

CJMH

to

n

M

CC CO CO c 0) W c CC <D CO 0 0 to e X u •H H c -p

<x CO 4J tJ> 0 •H (D CC CC 0 CD •H g X < 0 CO < OT Czj a g 4J c X C-H c X £ 0 0 u b 0 H crt CJ> c •H

<c *r^ c (i-, CC sz •H Jj

E-h Dli 4-> (U 0 0 0 0) to 2; OJ D CC 0 CO ISl Cd w 0 CT -U Dl, £ S-, (D •H s CO •H (U c X 0) 3 bO c H X CO (D CO 0 C to M a c 4-> CO to CO bO s Q. 0 X s >> 0) X s- Cd < 0 Cd Cd c/0 CC 0 0 0 H H

CO • • • CO t • • >H LO ^ >-l CO CT^ 0

QC to CM CM CM CO CM CM 00 M

CO • • X c X 0 w •H CO X 3 CO P

CO 00

X c 3 X 3 K CC 0 3 3 0

CO 0 0 Eh

X 00 •H

X P •fH CJ H CO H CJ C CD 3 > X CO c X M U (U 0 <=C 3 *H P 3 Eh X 3 •H H <c •H =c X pi C X 0 E P

CO Cl. 4->

X

C -H •H

X Eh 3 c M *H J 0 1—

0 X CJ > 3

CC Cd

J •p 0 •H ’H

X 3 c M «c

X

Cd

K <u 0 w c X P P 3 3 2 X 0 1—

H X *H 2 C-. CD Eh 3 *H c C > 1. 0 •H t. 2 CD E 2 CO 3 0 0 3 2 Eh c E bfl 3 < CJ CO X 0 CO > 4-> 0 •H C •H P P 0 2 M 3 C X

CC

X CO 0 •H CJ c 0 3 3 C- 1—

X P •H 3

<c CC JCX rH

C E 3 a •H 3 0 Eh s J- P p c X E-i CO CO w •H t- p= Eh bO 3 0 13, CJ 2 3 0 3 3 0 CO CO CO X 3 0 0 U 3 3 •H 3 .a: 0 3 J- 3 SL H H CO H CJ X 0 CC Q s X X X X X C-- Eh

CO btS s X

w

M •9 CO CO • « c • • • 0 « • . Eh > • . • Cd < -=r w Lf^ M3 X CO 0 2 2 r— X m =r O T— H r— T— X V* T— X M X X X X X 0

ffi • • • • X 3 3 E E 3 3 3 P P P 3 3

>. 3 >, to 3 3 C CO 3 0 c X C P 0 G P C •r4 0 0 3 •rH c. 0 bO P 2 •rH CO P •H 3 •rH E •rH 3 < P 3 3 P P 3 3 2 S 3 X P P 3 P 3 P 3 X c i- 0 •rH Eh X 0 3 3 3 Q 2

00 0 0 3 E •H

C •rH >, CJ X •rH X c •H XJ P P 0 CO X CO 3 2 M X P 0 0 X C 3 P •rH X •H P C

Eh

X 3 3 •rH CO \ 3 C •rH P 0 3 3 P 3

CO

X E J-, P P X 3 CO 0 s 3 p 3 P M •H 3 3 0 0 3 c n -H N CJ X P X

C

X

Q Eh

XX 3 Eh •rH 0 X P 3 •rH •a: CO 0 c X 2 0 P C CJ P •rH 3 3 3 c X bO •rH Eh <5 3 3 C •=c •rH P 0 p 3 3 3 c >. P 0 3 0 0 X P 3 c- 3 3 bO 2 •H P 3

<C 2 C 4^ 0 \ 0 •rH P CJ L.

X t, 0 P p rH

G

CC 0 0 •H P t. >H P 0 t. J- 0 P •rH 3 G 3 1—

M 00 O4 r-| P 3 3 X 3 3 i- 3 3 P X p 3 > 3 X Eh X 3 3 3 X P «C X X 3 P P •o -a: 3 3 3 0 >

CJ •a: 0 3 3 0 C

CQ 3 X 3 C

C c p 2; 2 2 X

X Eh

X CJC

U

MM 2 (J

X 2 P P 3 XX 2 X u X <c . . • CJ X «a; • • . . . 0 . . • Q • 0 T CM M 0 X X (X ^ IP X P IP CO a; CTi ^

r— t

COX • • • Q

FIGURE

7.1

THE

MONITORING

FRAMEWORK
444

• User conmitment characteristics — Factors related to user acceptance and support of the system.

While this structure is specifically aimed at the development circumstan

ces of the ETIP environment, it also reflects to some extent the types

of problem areas discussed in the literature from other experiences.

Finally, the monitoring framework has been illustrated with several system development cases from ETIP. First, three ETIP projects

have been briefly described and then used to provide examples of the

framework factors in actual developments. Second, an additional case,

that of the evaluation system project, has been described and used

to demonstrate how the framework as a whole might be applied. This

application further showed that the factors were relevant to an actual

development and that the framework could have assisted project managers in the earlier identification of one specific problem which arose

in the case. The application did not, however, provide supportive

illustrations of the strategy formulation, research, documentation,

or dissemination functions for which the framework has also been designed.

7.2 CONTRIBUTIONS OF THE RESEARCH

7.2.1 Increase in Confidence

The main claim made as a result of the research is that use of

the monitoring framework can help project managers more effectively

develop a complex system. Two requirements had to be met in order
445

to reach this conclusion. First, a management problem occurring with

the development of complex systems needed to be defined. Second,

a credible approach to the problem had to be developed. These requirements have been met to a varying degree by the research discussed

in this dissertation.

The following sections discuss the research results in relation

to these two requirements and the increases in confidence these results represent.

7. 2. 1.1 Definition of the Problem

The problems associated with developing complex systems are quite

extensively examined in the broad systems literature currently available. As shown in Chapter 3i numerous perspectives and sources exist

on these problems. Even though a broad study was made of the literature, the results presented in Chapter 3 by no means completely cover

the breadth and depth of the knowledge available.

The problems associated with managing complex developments are

thus, in one form or another, available in the literature. Even the

problem of monitoring a complex development to which the dissertation

is devoted is not a new concept. As shown in Chapter 4, several sources

in the literature have identified the need for monitoring approaches

which enable developers to control the numerous, intertwined factors

important in a complex project. Some writers have also suggested
the need for frameworks to structure the important factors in systems

development, although these proposals have not necessarily been aimed

at the monitoring problem in actual projects. It also seems likely,

since complex systems have been successfully produced in both public

and private sectors, that the monitoring problem has been recognized

before. This may be especially the case for the large scale systems

produced in the private sector and in military weapon system applications. These areas admittedly need a more extensive examination than

is provided in Chapter 3.

However, within the confines of ETIP experiences and the broad

range of literature which has been examined, an explicit, detailed

discussion of the monitoring problem (or approaches) has not been

found. In particular, existing presentations in the literature have

only generally raised the problem and have neglected to discuss its

dimensions or components. It appears overall that the various management problems of developing complex systems are not yet fully integrated with the numerous and more narrow facets of system development

that are extensively represented in the literature. This has especially been the case with the monitoring problem. The dissertation

has thus presented a more detailed examination of the problem than

previously available and has tied together many of the existing ideas

that are related to it.
447

Describing the Problem in the Dissertation

Definition of the problem was an exploratory process conducted

over a lengthy period of time. It involved detailed studies of ETIP

projects and the systems literature and repeated comparisons of these

sources to the needs of ETIP managers. During this process, the monitoring problem definition changed considerably.

To some extent, this exploratory process has been represented

by succeeding chapters of the dissertation. For example, the case

studies of Chapter 2 were written early when the monitoring problem

was still unclear. At that time, discovering factors important in

a development was the primary thrust of the research. These factors

were of course later adapted into the revised monitoring framework.

In a similar manner, Chapter 3 represented the continuing search for

factors after the case studies were completed. However, Chapter 3

also documented several key pieces of the evolving monitoring problem

which were uncovered by that time.

Because definition of the problem was a key contribution of the

research, it was decided that the process should be summarized in

one place (a case study of its own) . A portion of Chapter 4 was therefore used to trace development of the problem and show how the preceding background chapters had been a part of the process. Besides summarizing the problem and the methods used to define it. Chapter 4 also
448

enables readers to trace the process for themselves and to evaluate

the change in knowledge which has been achieved.

7. 2. 1.2 Development of the Monitoring Framework

Many of the ideas which have been presented for the monitoring

framework are neither novel to ETIP nor the literature. As shown

in the preceding chapters, most of the factors in the framework are

recognized in the systems literature or are a result of ETIP based

experiences. Organization of the factors into design, process, and

user commitment categories is not unique to the research either.

These terms and their corresponding elements are recognized in one

form or another in the literature, although the emphasis on user commitment in this research is more a result of ETIP thinking than a well

defined position in the literature. Finally, to some extent, even

the problem of monitoring has been discussed elsewhere.

However, it is the combination of these various ideas into a

specific approach that represents the contribution of this research

(besides a more explicit recognition of the problem, as discussed

above). The proposed framework offers a substantially more detailed

structure for monitoring than available and provides a mechanism of

value both to administrators of developments and to researchers of

system development processes.
449

An initial attempt has also been made to demonstrate the usefulness of the framework in an actual case. The application to the ETIP

evaluation system project showed that the framework factors were relevant to the case and that use of the framework as a whole might have

contributed to problem identification during a selected period.

The application has only been partially successful in demonstrating usefulness. Several problems in the test were noted:

• Some interaction between framework development and the specific case.

• Only one specific situation was explored with the framework.

• Not all functions of the framework were supported by exam- ples from the application.

• The application has not included comments or perspectives

from the personnel involved with the case.

Thus the usefulness of the framework, and ultimately its contribution

to the management and research of complex system developments, still

remains to be fully shown. Several ideas on further research in this

direction are presented in the concluding section of this chapter.

7.2.2 Utility of the Results

The results of the dissertation are useful to several audiences.

For ETIP, the identification of the monitoring problem and the studies

of specific projects represent the first detailed examination of system development processes within the program. In addition, the resulting monitoring framework is specifically aimed at ETIP projects
450

and offers managers an approach which should improve the conduct of

developments and the chances of success. The dissertation thus documents important lessons of the ETIP research program and using these

1 ideas provides a technique for future projects.

For others outside of the ETIP environment, the results may have

both administrative and research value. Other system developers may

find that the framework offers an approach to improving the management

of complex system developments in their own situations. For instance,

the framework factors can highlight the various factors which may

be important in their development projects. Alternatively, the functions monitoring can serve may suggest ways that a structured, routine

information gathering process can increase the manager’s chances of

success. It is not being suggested, however, that the framework as

a whole can replace the seasoned judgement of experienced developers.

Rather, it is the overall thrust of the monitoring approach, which

can easily be adapted to different situations, that other developers

2

may find to be the most significant contribution.

For researchers interested in system development processes, the

results of the dissertation present several useful contributions.

Some comments on use of the monitoring framework within ETIP have

been obtained. See APPENDIX C.

The approach may be most useful when tied to a front-end system plan

ning process which identifies in advance the key areas of concern

in a project (Parsons, Note 1). See APPENDIX C.
First, the identification of the monitoring problem represents a more

detailed examination of the problem than has been previously undertaken in most parts of the literature. The general problem of monitoring which several sources present has been related to other ideas

on important development problems, thus providing some dimensions

and elements to the problem.

Second, the supporting background reviews of the dissertation

offer researchers several items which may be useful in the structure

of future research. The first of these is the broad structuring of

the current literature into four distinct areas (case studies, models

of development, important factors in development, and system designs

arising from specific projects). It has been shown that the distinct

perspectives of these areas have yet to be commonly combined into

research endeavors, thus narrowing the contribution researchers can

make by relying on one perspective or another. The use of each perspective in this dissertation (i.e., combining case studies, models,

factor, and design approaches) may offer researchers an alternative

on how to improve their presentations and make them more usable by

others. A second item useful to researchers may be the example case

studies which have been used to describe specific ETIP experiences.

Detailed reviews of projects similar to these are not commonly found

in the systems research literature. Thus, the cases presented may

offer a model that in the future can be replicated in many other situations, thereby increasing the exposure of readers of the literature

to what actually happens.
Finally, the framework offers researchers a mechanism which may

assist in the design and implementation of future studies. The monitor

ing process provides researchers with a means of conducting research

on projects in real-time. By combining research objectives with administrative use of the framework, researchers may be able to design and

implement studies in actual projects rather than relying solely on

retrospective approaches or more informal case studies. This may,

in turn, help generate the empirical evidence that several systems

researchers identify as needed.

More generally, the results of the dissertation may be of use

to developer and researcher audiences involved with a wide variety

of complex systems. While the framework has been specifically designed

for the information system projects of the ETIP environment, it has

relied on concepts endogenous to both hardware and software systems,

and public and private experiences. System development problems are

similar in these areas, but they have not often been compared. In

addition, useful approaches to development from these areas have often

not been combined. The dissertation may thus help promote the transfer of knowledge between these sectors and offer guidance to developers regardless of the type of system being developed.

7.3 EXTENSIONS OF THE RESEARCH

There are several problems which remain unresolved in the disserta

tion and these suggest several areas where further research may be
conducted. These problems center upon two areas: framework design

and testing. The following sections discuss each area.

7 . 3 •! Improving Framework Design

The current framework design represents the initial attempt by

the writer to define a monitoring structure for the management of

system developments. Since the framework is preliminary, it is likely

that there are gaps in the factors and that alternative organizations

or structures might improve its usefulness and comprehensiveness.

Concerning the framework factors, several possible changes have

already been identified. For example:

• The subcategory of Operation and Performance factors might

be separated into two areas, thus increasing the attention

on each (Libman, Note 2). In addition, a factor for maintenance of the system might be added to the operation factors.

• Concerning Team Personnel Factors, two other factors, motivation and recruitment of personnel, might be added as separate items (Thompson, Note 3)*

• The term Task Size might be better phrased. In addition,

a factor concerning project goals might be added in order

to highlight the more overall task structure.

• A factor for planning might be added to Process Characteristics

although it is unclear where it should be placed since elements of it are present in several of the existing factors

(Thompson, Note 3)*

• Requirements analysis may need more emphasis in the framework, possibly under the Problem Identification factor (Berlin,

Note 4).

• There may be alternative names to the factors in System

Use Factors that are better designed.
454

Finally, for all of the factors, the brief descriptions and the support3 ing discussions might be improved.

Concerning the organization of the framework, the categories

used above the factor level, such as Task Structure Factors, may also

need revision. These categories were originally included to highlight

closely related factors and to facilitate understanding the numerous

factors. However, it may be that framework users won’t pay much attention to them (Thompson, Note 6) and that their function is limited.

A simplified structure may be more useful, although it is currently

unclear what this might contain.

Ideas for either better factors or organization may be available

from other experiences which haven’t been reviewed for this research.

For example, monitoring approaches may be available from experiences

in the private sector or in military systems applications. It would

also seem likely that techniques used in sales and marketing desciplines would be applicable (Berlin, Note 4). These areas need more

examination

In addition, actual use of the proposed framework might uncover

better ideas. Some thoughts on testing the framework are discussed

below.

3 A related problem is that the text is too long to promote applications

among readers (Schofer, Note 5). A manual summarizing the monitoring

process proposed in the dissertation has thus been developed and is

presented in APPENDIX B.
1 . 3»2 Testing the Framework

Perhaps the most significant problem of the research is that

the framework remains untested in an actual system development. The

retrospective application of the framework to the ETIP evaluation

system project has not been a very effective substitute for a real

application. It has at best suggested that the framework might be useful to project managers in the solution of complex problems arising in

a development. It has not shown that the framework might also be

useful to managers for strategy development, dissemination, or documentation, and useful to researchers for the conduct of structured

investigations of development processes.

In developing an actual test, several questions about the framework should be considered. The first of these is to determine whether

the monitoring functions are useful to project managers. This would

include, the example, whether the framework is a useful mechanism

in identifying problems that are of interest and importance to project

managers. Generally, a future test of the framework should examine

questions similar to this for each function. Other suggestions are

as follows:

• Problem Identification.

Is the framework useful in identifying problems, both in

the short and long term?

• Strategy Development.

Is the framework useful in helping to revise strategy?

Does it provide an information base that can help devise

strategy? Does it provide a means of evaluating strategy?
456

• Research.

Does the monitoring framework enable research to be conducted on an evolving development? Are there areas where researchers can support project managers? How should monitoring

be conducted to help improve research designs? Does the

process provide a useful means of information transfer between developments and the research literature?

• Documentation.

Does documentation provide a means to identify special,

evolving, or recurring problems not evident on a short term

basis? Does documentation help evaluate strategies and

provide decisionmaking information? Does documentation

provide a means of informing new staff and help alleviate

the problem of turnover?

• Dissemination.

Does the framework monitoring make it easier to generate

key information for those inside and outside a project?

Does this dissemination help promote coordination? Does

monitoring information itself help facilitate further implementation of the monitoring process as a project unfolds?

A second line of questions for future framework testing should

be aimed at whether the framework factors are appropriate for the

functions. For example, it must be examined whether problems that

arise in a development can be adequately characterized and analyzed

by the existing factors. Analysis would have to include how the factors are used, whether there are any missing or inappropriate factors,

or whether they might be better organized. Examination at this level

should help determine what revisions are needed in the current structure .

In a broader perspective, future testing of the framework should

also examine whether the framework is of value to administrators and

researchers of different types of complex system developments.
457

REFERENCE NOTES

CHAPTER 2

1. Experimental Technology Incentives Program. Design and use of

life cycle cost models for the planning and acquisition of federal

space . Unpublished project plan for project #36, May 1974.

2. Public Buildings Service. Design and use of life cycle cost models

for the planning of federal space . Unpublished preliminary draft

of a project plan, March 1974.

3. Mar is cal and Company. A proposal to develop a life cycle planning

and budgeting model through expansion of contract (GS-00-8-02047)

September 1974.

4. Ostrander, V. Personal communication, February 24, 1978.

5. Galuardi, J. F. Letter to J. D. Lewis, ETIP Director, April

29, 1974.

6. Penn, R. T. Personal communication, November 20, 1977.

7. Sampson, A. F. Letter to J. D. Lewis, ETIP Director, May 24,

1974.

8. Timbers, M. J. Letter to J. D. Lewis, ETIP Director, June 11,

1974.

9. Holding, W. Letter to J. D. Lewis, ETIP Director, May 9, 1974.

10. Experimental Technology Incentives Program. Systematizing

local procurement as the preface to technological change . Unpublished project plan for project #75, November 1974.

11. Experimental Technology Incentives Program. Public market

aggregation as an incentive for technological change . Unpublished

project plan for project #60, June 1974.

12. Cornett, R. M. Letter to R. J. Barra, ETIP Staff, December

7, 1973.

13. Cornett, R. M. Purchasing power and technological change , phase 1 - draft final report . Unpublished draft report to ETIP,

June 30, 1976.

14. Wagner, M. , and Zeldis, K. An analysis of the intervention

into state government procurement by the experimental technology

incentives program . Unpublished final report, July 1977.
458

15. Berke, J. G. Letter to R. M. Cornett, National Association

of State Purchasing Officials, August 7, 1975.

16 . Berke, J. G. Meeting on the development of an information system , for procurement specifications, policies, etc . Memorandum to

R. M. Cornett, L. Spangler, J. McElwee, C. Travis, G. Rowland, and

T. Fody, July 1, 1975.

17. Berke, J. G. Personal communication, October 21, 1976.

18. Berke, J. G. State and local experiment strategy . Draft

paper, November 11, 1976.

19. Cornett, R. M. Personal communication, March 9, 1977.

20. Spangler, L. Letter to J. D. Lewis, ETIP Director, March

23, 1977.

21. National Institute of Government Purchasing, Inc. Untitled

paper submitted to ETIP, March 1977.

22. O'Conner, M. Personal communication, January 26, 1977.

23. Arnold, J. Personal communication, February 1, 1977.

24. Lewis, J. D. NIGP . Memorandum to T. Fody, ETIP Project

Staff, October 24, 1974.

25. Lewis, J. D. Letter to R. M. Cornett, National Association

of State Purchasing Officials, and L. Spangler, National Institute

of Governmental Purchasing, Inc., February 1, 1977.

26. Lewis, J. D. Letter to R. M. Cornett, National Association

of State Purchasing Officials, April 12, 1977.

27. Lewis, J. D. Letter to L. Spangler, National Institute of

Governmental Purchasing, Inc., April 8, 1977.

28. Hulick, C. H. Personal communication, November I6 , 1979.

29. Hall, A. H. Letter to J. D. Lewis, ETIP Director, December

23, 1974.

30. Experimental Technology Incentives Program. Systematizing

local procurement as the preface to market aggregation . Draft

project plan for project #75, September 17, 1974.

31. Experimental Technology Incentives Program. Notes on the

ETIP Symposia, workshops 2A and 28 "Interaction and Information

Exchange," January 1975.
32. Experimental Technology Incentives Program. Competency evaluation of small R&D firms . Unpublished project plan for project

#46, June 1974.

33. Charney, T. Personal communication, September 23, 1977.

34. Innovative Systems Research. Preliminary plan-Draft . Preliminary draft report to ETIP, December 23, 1975.

35. Innovative Systems Research. Preliminary plan . Unpublished

report to ETIP, January 30, 1976.

36. Innovative Systems Research. Test report . Unpublished report

to ETIP, September 24, 1976.

37. Braudy, R., and Albert, S. Personal communication, October

11, 1977.

38. Penn, R. T. Personal communication, January 15, 1977.

39. Innovation Systems Research. Final report - PDQ program . Unpublished report to ETIP, December 1977.

40. Innovative Systems Research. PDQ manual system . Unpublished

report to ETIP, December 1977.

41. Innovative Systems Research. Users manual - Computerized

PDQ system . Unpublished report to ETIP, December 1977.

42. Innovative Systems Research. Data encoding manual for the

computerized PDQ system, CSSBLM . Unpublished report to ETIP,

December 1977

43 . Penn, R. T. Personal communication, August 12, 1977.

44. Charney, T. Personal communication, June 21, 1978.

45 . Small Business Administration, Technology Assistance Division.

Procurement Automated Source System (PASS). Undated paper.

CHAPTER 3

1. Taggart, W.M. , and Tharp, M. 0. A survey of information require - ments analysis techniques (Working Paper 76-1). Unpublished

manuscript, Florida International University, 1976.

2. Duchesneau, T.D. Determinants of adoption of a major innovation: The role of economic and organizational factors . Paper

presented at the ORSA/TIMS National Conference, Miami, Florida,

November 3-5, 1976.
460

3. Jue, S.J., Nowocin, R.V., and Mandelbaum, C. R. A systematic

approach to procuring and developing computerized models . Paper presented at the ORSA/TIMS National Conference, Miami,

Florida, November 3, 1976.

4. Mitroff, I. I., and Emshoff, J.R. On strategic assumption- making: A dialectical approach to policy and planning . Unpublished manuscript. University of California, Berkeley,

no date.

5. Rich, R.F. Management and problem solving styles: An assessment

of information system designs . Unpublished manuscript, Princeton

University, 1978.

6. Thompson, C.W.N. Personal communication, October 23, 1978.

7. Nay, J.N., Scanlon, J.S., Graham, L. , and Waller, J.D. The

National Institute's information machine, a case study of

the national evaluation program (contract report no. 9-5070-05).

Washington, D.C.: The Urban Institute, August 15, 1977.

8. Weiss, R.G., and Thompson, C.W.N. Improving the likelihood

of implementing program results . Unpublished manuscript, Northwestern University, 1978.

9. Williams, W. Developing an implementation perspective . Paper

presented at the National Convention of the Western Political

Science Association, March 23, 1979.

CHAPTER 4

1. Frederick, W. A. Personal communication, July 20, 1978.

2. Thompson, C. W. N., Libman, A. S., and Garrity, S. D. Methods

used in the acquisition and development of evaluation and informa- tion systems . Paper presented at the ORSA/TIMS National

Conference, New Orleans, Louisiana, April 30 - May 2, 1979.

3. Taggart, W. M. , and Tharp, M. 0. A survey of information require - ments analysis techniques (Working Paper 76-1). Unpublished

manuscript, Florida International University, 1976.

4. Williams, W. Developing an implementation perspective . Paper presented at the national convention of the Western Political Science

Association, March 23, 1979.
CHAPTER 5

1. Henderson, M. System development-Overview ETIP workshop

paper . Unpublished working paper for ETIP, August 1976.

2. Keen, P. Presentation at a session. "Implementation Problems

and Strategies," in national conference of the American Institute

for Decision Sciences, Chicago, Illinois, October 19, 1977.

3. Ginzberg, M. Presentation at a session, "Implementation Problems

and Strategies," in national conference of the American Institute

for Decision Sciences, Chicago, Illinois, October 19, 1977.

4. Rice, D. Presentation about a defense resource management study

at a meeting of the American Society for Public Administration,

Rosslyn, Virginia, March 13, 1979.

5. Thompson, C.W.N., and Rath, G. J. Making your health system

work. Paper presented at the Annual Meeting of the Academy of

Pediatrics, Chicago, Illinois, October 20-24, 1973.

6. Jue, S.J., Nowocin, R.V., and Mandelbaum, C.R. A systematic

approach to procuring and developing computerized models . Paper

presented at the ORSA/TIMS National Conference, Miami, Florida,

November 3, 1976.

7. Young, C.J. Evaluation utilization . Paper presented at the

Second Annual Meeting of the Evaluation Research Society,

Washington, D.C., November 2-4, 1978.

8. Libman, A.S. Personal communication, July 12, 1979.

CHAPTER 6

1 . COTR . COTR’s specific comments on commercial impact project

report, 1st qtr. (continued ) . COTR comments on the first quarter

report from Contractor C, November 10, 1976.

2. ETIP. Notes on a meeting in Experimental Methods with the COTR,

July 8, 1976.

3. ETIP. Agenda . Draft of agenda items for ETIP - Contractor C

meeting of July 28, 1976, July 22, 1976.

ETIP. Agenda . Draft of agenda items for ETIP - Contractor A

meeting of July 13, 1976, July 8, 1976.

4 .
462

5. COTR. Contractor A - Contractor C- PAAs and ETIP . Notes on a meeting with Contractor A, Contractor C, several of the PAAs,

and ETIP on August 10, 1976, August 11, 1976.

6. ETIP. Annotated agenda . Notes for ETIP - Contractor A meeting

of July 13, 1976, July 12, 1976.

7. ETIP. Policy on dummies . Notes on ETIP policy concerning the use of dummies in the project, August 10, 1976.

8. Contractor C. Minutes of ETIP meeting - July 28 and 29, 1976 . Minutes of ETIP, Contractor C, Contractor A, and PAAs meeting

held at ETIP (July 28, 1976) and at Contractor C (July 29,1976)

offices, August 6, 1976.

9. COTR. Contact report . Notes on telephone call with Contractor

A on September 22, 1976, October 1, 1976.

10. ETIP. Contractor C - ETIP, July 28, 1976 . Notes on tape of

Contractor C and ETIP meeting held on July 28, 1976, July 28,

1976.

11. ETIP. First Contractor A meeting . Notes on ETIP - Contractor

A meeting of July 13,1976, July 13, 1976.

12. ETIP. Letter to Contractor C with comments on first dummy report

of first quarter reports, August 31, 1976.

13 . ETIP. Notes on ETIP - Contractor A meeting of July 13, 1976,

July 13, 1976.

14. Contractor A. Design and development of evaluation systems to

measure the agency impact of a series of procurement experiments

being conducted in cooperation with specific government purchas- ing agencies . Contractor A technical proposal to ETIP for con- duct of agency impact evaluation, April 15, 1976.

15 . Contractor C. Design and development of an evaluation system

to measure commercial impacts of a series of procurement experi- ments . Contractor C technical proposal to ETIP for conduct of

commercial impact evaluation, April 15, 1976.

16 . Contractor A. A draft list of the major deliverables and responsible staff for Phase 1, July 29, 1976.

17 . Contractor C. Deliverables for the first quarter . A list of

deliverables for the first quarter and the staff responsible

for their production, July 23, 1976.
463

18. Contractor C. Letter to COTR at ETIP containing monthly progress

report No. 2 for Contractor C for August, August 13, 1976.

19. Contractor C. First quarter deliverable . First quarter report

from Contractor C, October 15, 1976.

20. Contractor A. First quarter report . First quarter report from

Contractor A, October 1, 1976.

21. Contractor A. ETIP project activity summary, agency impact , October 1976 . Summary list of Contractor A activities in October

1976, no date.

22 . COTR . Notes on Contractor A*s first set of formal reports, de- livered September 30, 1976 . COTR comments on the first quarter

report from Contractor A, October 4, 1976.

23. ETIP. Notes on ETIP - Contractor C meeting on November 18, 1976,

November l8, 1976.

24. ETIP. Contact report . Notes on telephone conversation with

Contractor A on November 3, 1976, November 17, 1976.

25. COTR. Contact report Notes on ETIP - Contractor C meeting

at ETIP, September 16 , 1976, September 17, 1976.

26. ETIP. Notes on telephone conversation with Contractor A, December

21, 1976.

27. Tape of Meeting. Tape of ETIP - Contractor C meeting on November 18, 1976.

28. COTR. Contact report . Notes on conversation with Contractor A,

September 9, 1976.

29. COTR. Notes on a meeting between staff of Experimental Methods,

July 22, 1976.

30. Transcript of Meeting. Contractor A meeting, July 20, 1976 . Transcript of ETIP - Contractor A meeting on July 22, 1976,

no date.

31. Contractor C. Record, ETIP procurement meeting, August 19, 1976 . Notes from Contractor C on ETIP-Contractor C meeting of August 19,

1976, no date.

32. Contractor C. ETIP/Contractor C/Contractor A, Wednesday session.

Notes from Contractor C on ETIP - Contractor C - Contractor A

meeting on August l8 , 1976, August l8 , 1976.
464

33* Contractor C. Wednesday p.m . Notes from Contractor C on late

Wednesday portion of ETIP - Contractor C - Contractor A meeting

on August l8, 1976 , August I 8 , 1976.

34 . ETIP. Contact report . Notes on telephone conversation with

Federal Supply Service staff, November 5, 1976.

35 . ETIP Notes on Experimental Methods - Procurement Program meeting, August 30 , 1976.

36 . COTR. Notes on two telephone conversations with Contractor C on September 1, 1976, September 2, 1976.

37 . ETIP. Notes on telephone conversation with Charles W. N. Thompson,

July 7, 1976.

38 . ETIP. Agenda, Contractor C critical review session, first quarter

report, October 28, 1976 . ETIP notes on issues to be discussed

at ETIP - Contractor C meetings on October 28, 1976, October 27,

1976.

39 . COTR. Action needed/General comments and commitments made during

Contractor C first quarterly critical review session . COTR notes

on videotape of ETIP - Contractor C meeting on first quarter

reports on October 28, 1976, October 31, 1976.

40. Contractor C. Letter to COTR, ETIP, reviewing agreements reached

at ETIP - Contractor C meeting of November I 8 , 1976, November 24,

1976.

41. COTR. Letter to Contractor C providing COTR’s 'stakeholder'

comments on a Contractor C report, December 1, 1976.

42. COTR. Contact report . Notes on telephone conversation with

Contractor C on September 24, 1976, October 1, 1976.

43 . Notes on ETIP - Contractor C meeting of August 28, 1976.

44. COTR. Contact report . Notes on telephone conversation with

Contractor C on September 8, 1976, September 9, 1976.

45 . Contractor C. First quarter deliverable of reports and liaison . Portion of first quarter report from Contractor C, October 15,

1976.

46. ETIP. Contact report . Notes on telephone conversation with

Federal Supply Service staff on October 12, 1976, October 12,

1976.
465

47. Contractor C. Revised commercial impact evaluation management

plan . December 15, 1976.

CHAPTER 7

1. Parsons, R.E. Letter to C.W.N. Thompson, April 10, 1981.

2. Libman, A.S. Personal communication, August 1980.

3. Thompson, C.W.N. Personal communication. May 24, 198O.

4. Berlin, V.B. Personal communication, April 12, I98I.

5. Schofer, J.L. Personal communication, April 13, 1981.

6. Thompson, C.W.N. Personal communication, March 9, 1981.
466

BIBLIOGRAPHY

Ackoff, R. L. Unsuccessful case studies and why. Operations

Research , I960, 8 ( 2 ), 259-263.

Ackoff, R. L. Management misinformation systems. Management

Science , 1967, j^(4), B147-B156.

Air Force Systems Command. System engineering management procedure

(AFSCM 375-5). Washington, D.C.: March 10, 1966.

Alexander, M. J. Information systems analysis; Theory and applica- tions . White Plains, New York: Science Research Associates,

1974.

Alter, S. A taxonomy of decision support systems. Sloan Management

Review , 1977, 19(1), 39-56.

Alter, S. , and Ginzberg, M. Managing uncertainty in MIS implementation. Sloan Management Review , 1978, ^(1), 23-31«

Anderson, J. , and Narasimhan, R. Assessing project implementation

risk: A methodological approach. Management Science , 1979, 25(6),

512-521.

Anthony, R. N. Planning and control systems: A framework for

analysis . Boston: Harvard University, 1965.

Argyris, C. Management information systems: The challenge to

rationality and emotionality. Management Science , rZ(6), B275-

B292.

Arnovick, G. N., and Gee, L. G. Design and evaluation of information systems. Information Processing and Management , 1978, l4(6 )

369-380.

Blumenthal, S. C. Management information systems: A framework for

planning and development . Englewood Cliffs, New Jersey:

Prentice-Hall, 1969.

Boland, R. J. The process and product of system design. Management

Science , 1978, ^(9), 887-898.

Brown, R. V., and Watson, S. R. Pre-testing innovation: Methodology for testing and the design of management systems. Theory

and Decision, 1977, 8(4), 315-336.
Campbell, D. T. Administrative experimentation, institutional

records, and nonreactive measures. In J.C. Stanley (Ed.), Improving

Experimental Design and Statistical Analysis . Chicago: Rand

McNally, 1967.

Chen, S., and McCallum, C. J. The application of management science

to the design of telephone directories. Interfaces , 1977, Part 2,

8(1), 58-69.

Chervany, N. L. Management of MIS implementation: A parallel

to the traditional life cycle model. Proceedings of the 10th

Annual Meeting of AIDS , 1978, pp. 176-178.

Chestnut, M. Systems engineering methods . New York: Wiley,

1967.

Churchman, C. W. The philosophy of experimentation. In 0. Kempthorne,

T. A. Bancroft, J.W. Gowen, and J. L. Lush (Eds.), Statistics

and mathematics in biology . Ames, Iowa: Iowa State College Press,

1954.

Churchman, C. W. and Schainblatt, A. H. The researcher and the

manager: A dialectic of implementation. Management Science , 1965, 11(4), B69-B73.

Comptroller General of the United States. HUD*s evaluation system —An assessment (PAD-78-44). Washington, D. C.: July 20, 1978.

Comptroller General of the United States. Ways to improve manage- ment of Federally funded computerized models (LCD-75-111). Washington,

D.C.: 1975.

Davis, G. B. Management information systems: Conceptual foundations, structure, and development . New York: McGraw-Hill, 1974.

Dietrich, F. Systems acquisition: How A- 109 can help shorten

the process. Government Executive , 1977, £(10), 10-12.

Edelman, F. They went thataway. Interfaces , 1977, 7(3), 39-43.

Edelstein, M. , and Melnyk, M. The pool control system. Interfaces , 1977, 8(1), Part 2, 21-36.

Ein-Dor, P., and Segev, E. Organizational context and the success

of management information systems. Management Science, 1978,

24(10), 1064-1077.

Eveland, J. D., Rodgers, E.M. , and Klepper, C. The innovation

process in public organizations (NSF/PRA-7517952/1/7) . Ann Arbor:

University of Michigan, March 1977. (NTIS No. PB-266 234)
Flagle, C. D., Huggins, W. H., and Roy, R. H. Operations research

and systems engineering . Baltimore: Johns Hopkins Press, I960.

Flaherty, K. R. Innovation in state public utility commissions : An exploratory study of techniques in energy regulation . Washington,

D.C.: Experimental Technology Incentives Program, National Bureau

of Standards, 1980. (NBS No. NBSIR 80-2046)

Fronk, W. C. Solving five common systems development problems.

Journal of Systems Management , 1978, 29(7), 34-37.

Fudge, W. K., and Lodish, L. M. Evaluation of the effectiveness

of a model based salesman’s planning system by field experimentation. Interfaces , 1977, ^(1), Part 2, 97-106.

Gall, J. Systematics: How systems work and especially how they

fail . New York: Quadrangle/New York Times Book Company, 1977.

Ginzberg, M. J. Steps towards more effective implementation of

MS and MIS. Interfaces , 1978a, 8(3), 57-63.

Ginzberg, M. J. Finding an adequate measure of OR/MS effectiveness. Interfaces , 1978b, 8(4), 59-62.

Gordon, H. J. The role of the contract in systems acquisition.

Defense Systems Management Review , 1980, 3.(1), 30-42.

Gorry, G. A., and Morton, M. S. S. A framework for management

information systems. Sloan Management Review , 1971, j3(D, 55-70.

Hall, A. D. A methodology for systems engineering . Princeton,

New Jersey: Van Nostrand, 1962.

Harrison, D. B. Owner oriented information systems for multiproject control. Project Management Quarterly , 1978, _^(2), 27-30.

Hartman, W. , Matthes, H. , and Proeme, A. Management information

systems handbook . New York: McGraw-Hill Book Company, 1968.

Haynes, R. H., and Wheelwright, S. C. Link manufacturing process

and product life cycles. Harvard Business Review , 1979, 57(1),

133-140.

Hill, L. S. System engineering in perspective. IEEE Transactions

on Engineering Management , 1970, EM-17 ( 4) , 124-131.

Ibrahim, R. L. Software development information system. Journal

of Systems Management, 1978, _^(12), 34-39.
Jones, M. M. , and McLean, E. R. Management problems in large

scale software development projects. Sloan Management Review , 1970, n( 3 ), 1-16.

Karger, D. W. , and Murdick, R. G. A management information system

for engineering and research. IEEE Transactions on Engineering

Management , 1977, EM-24 (2) , 72-75.

Katch, D. The making of MIS-conclusion . Infosystems , 1978, 25(8)

54-59.

Kay, R. H. The management and organization of large scale software

development projects. Proceedings of the Spring Joint Computer

Conference, American Federation of Information Processing Societies , 1969, 34» PP- 425-433.

Keen, P. G. W. , and Morton, M. S. S. Decision support systems:

An organizational perspective . Reading, Massachusetts: Wesley

Publishing Co., 1978.

King, W. R., and Cleland, D. I. Strategic planning and policy . New York: Van Nostrand Reinhold, 1978.

Kolb, D. A., and Frohman, A. L. An organizational development

approach to consulting. Sloan Management Review , 1970, 12(1)

51-65.

Kraemer, K. L. , and King, J. L. "Laissez-innover” : A critique

of Federal involvement in development of urban information systems.

The Bureaucrat , 1978, (7(3), 23-31.

Krasnican, M. J. Managing the development of an experimental

computer-aided technology planning system (PLANET). IEEE Transac - tions on Engineering Management , 1971, EM-l8 (2) , 50-57.

Kronenberg, R. A. Weyerhauser ’ s management information system.

Datamation , 1967, ]3(5), 28-30.

Lawrence, J. E. S., and Garrity, S. D. Agency impact analysis:

Life cycle costing in the Public Buildings Service . Washington,

D. C.: Experimental Technology Incentives Program, National

Bureau of Standards, May 1979. (NBS No. NBS-GCR-ETIP 79-68)

Lewin, K. Group decision and social change. In H. Proshansky and

B. Seidenberg (Eds.), Basic studies in social psychology . New

York: Holt, Rinehart and Winston, 1965.

Lewin, K. Frontiers in group dynamics. Human Relations, 1, pp. 2-38.
Libman, A. S. Procurement of evaluation systems; A case study

of the parametric factor evaluation approach to source selection . Washington, D.C.: Experimental Technology Incentives Program,

National Bureau of Standards, July 1980. (NTIS PB80-204654)

Livingston, J. S. Weapon system contracting. Harvard Business

Review , 1959, 37(4), 83-92.

Lodish, L. M. CALL PLAN: An interactive salesman’s call planning

system. Management Science , 1971, ]^(4), Part 2, 25-40.

Lodish, L. M. A vaguely right approach to sales force allocation

decisions. Harvard Business Review , 1974, ^(1), 119-124.

Lorerzen, G. L., and Braskamp, L. A. Comparative influence of

political, cost/benefit, and statistical information on administrative decisionmaking. Evaluation and Program Planning , 1978,

1(3), 235-238.

Lorsch, J. W. Making behavioral science more useful. Harvard

Business Review , 1979, ^(2), 171-180.

Lucas, H. C. Measuring employee reactions to computer operations.

Sloan Management Review , 1974a, _^(3), 59-67.

Lucas, H. C. Towards creative system design . New York: Columbia

University Press, 1974b.

Lucas, H. C. Performance and the use of an information system.

Management Science , 1975a, ^(8), 908-919.

Lucas, H. C. Why information systems fail . New York: Columbia

University Press, 1975b.

Lucas, H. C. The implementation of computer based models . New

York: National Association of Accountants, 1976.

Lucas, H. C. Unsuccessful implementation: The case of a computer

based order entry system. Decision Sciences , 1978a, ^(1), 68-79.

Lucas, H. C. The evolution of an information system: From keyman to every person. Sloan Management Review , 1978b, ]^(2), 39-52.

Lucas, H. C., and Plimpton, R. B. Technological consulting in

a grass roots, action oriented organization. Sloan Management

Review , 1972, ^(1), 17-36.

Malvey, M. P. Management information systems and the organization

of complexity: A hospital installs a seemingly simple system

(Doctoral dissertation. University of California, 1977). Disser - tation Abstracts International, 1978, 3£(2), 1080A.
Mann, F. C., and Williams, L. K. Observations on the dynamics

of a change to electronic data processing equipment. Administra- tive Science Quarterly , I960, 5 ( 2 ), 217-256.

Mason, R. 0., and Mitroff, I. I. A program for research on management information systems. Management Science , 1973 > 1^(5), 475-487.

Matthews, D. Q. The design of the management information system . New York: Auerbach Publishers, 1971

McGrath, T. Management information systems: It’s all in the

tailoring. Innovation , 1970, ]3> 44-49.

McNaugher, T. L. Marksmanship, McNamara, and the Ml6 rifle:

Innovation in military organizations. Public Policy , 1980, 28(1),

1-37.

Meador, C. L., and Ness, D. N. Decision support systems: An

application to corporate planning. Sloan Management Review , 1974,

]^(2), 51-68.

Miller, G. H., and Wilier, B. Information systems for evaluation

and feedback in mental health organizations. In L. Rutman (Ed.),

Evaluation research methods . Beverly Hills, California: Sage

Publications, Inc., 1977.

Mitroff, I., Williams, J., and Rathswohl, E. Dialectical inquiring systems: A new methodology for information science. Journal

of the American Society for Information Science , 1972, 23(6),

365-378.

Mohan, L. , and Bean, A. S. Introducing OR/MS into organizations:

Normative implications of selected Indian experience. Decision

Sciences , 1979, 10_(D> 136-150.

Moore, J. H. Effects of alternate information structures in a decomposed organization: A laboratory experiment. Management

Science , 1979 , 25 ( 5 ), 485 - 497.

Moore, L. T., and Byrd, J. For practitioners only: A case study

in selling an OR/MS project to a nameless company. Interfaces , 1977, 8(1), Part 1, 96-104.

Mumford, E. , Land, F., and Hagwood, J. Participative approach

to the design of computer systems. Impact of Science on Society,

1978, ;^(3), 235-254.

Munro, M. C. Determining the manager's information. Journal of

Systems Management , 1978, ^(6), 34-39.
472

Murdick, R. G., and Ross, J. E. Information systems for modern

management . Englewood Cliffs, New Jersey: Prentice-Hall, 1971.

Nadler, D. A., Cammann, C. T., and Mirvis, P. H. Developing a feedback system for work units: A field experiment in stuctural

change. The Journal of Applied Behavioral Science , 1980, 16(1)

41-62.

Nay, J. N. Evaluating reality: How a communication gap was closed in one urban community. Proceedings of the 1973 International Conference

on Cybernetics and Society, IEEE Systems, Man and Cybernetics Society , 1973, pp. 120-123.

Nicholas, J. M. Transactional analysis for systems professionals.

Journal of Systems Management , 1978, ^(10), 6-13.

Office of Federal Procurement Policy. Major system applications , a discussion of the application of 0MB circular No. A- 109 (OFPP

Pamphlet No. 1). Washington, D. C. : August 1976.

Palmer, R. P. Case studies in library computer systems . New York:

R. R. Bowker, 1973*

Patterson, M. B. Government-contractor adversarial relationships.

Defense Management Journal , 1977, J^(3), 57-62.

Peck, M. J., and Scherer, F. M. The weapons acquisition process : An economic analysis . Boston: Graduate School of Business Administration, Harvard University, 1962.

Peterson, E. L. MIL-STD-499A and its application to systems en- gineering. Defense Systems Management Review , 1980, 3 ( 2 ), 105-108.

Radnor, M. , Rubenstein, A. H. , and Tansik, D. A. Implementation

in operations research and R&D in government and business organization. Operations Research , 1970, 1^(6), 967-991.

Resnikoff, H. L. The need for research in information science.

Information and Management , 1979, 2 , 1-2.

Robey, D., and Zeller, R. L. Factors affecting the success and

failure of an information system for product quality. Interfaces , 1978, 8(2), 70-75.

Roseman, J. A. An exploratory research process model (Doctoral

dissertation. Northwestern University, 1977). Dissertation Abstracts

International , 1978, 39(8), 3820B.

Ruth, S. R. The advances logistics system: An idea whose time

had not quite arrived. Defense Management Journal , 1978, ^(2), 35-39.
473

Schein, E. H. Management development as a process of influence.

Industrial Management Review , 1961, 2 ( 2 ), 59-77.

Simon, H. A. The new science of management decision . New York:

Harper & Row, I960.

Smith, D. An organization for successful project management.

Proceedings of the Spring Joint Computer Conference, American

Federation of Information Processing Societies , 1972, 40, 129-140.

Sollenberger , H. M. Management control of information systems

development . New York: National Association of Accountants, 1971.

Taggart, W. M. , and Tharp, M. 0. A survey of information requirements analysis techniques. Computing Surveys , 1977, 9(4), 273-290.

Thompson, C. W. N. Implementation. Defense Transportation Journal , 1974, ^(1), 30-31.

Thompson, C. W. N. Technology utilization. In C. A. Cuadra (Ed.),

Annual review of information science and technology (Vol. 10).

Washington, D. C.: American Society for Information Science, 1975.

Thompson, C. W. N. Evaluation system proposal preparation and

evaluation procedure . Washington, D. C.: Experimental Technology

Incentives Program, National Bureau of Standards, February 1976.

(NTIS No. PB273666)

Thompson, C. W. N. The functional value of uncertainty in the

procurement process. Proceedings of the Seventh Annual Acquisition

Research Symposium , 1978a, pp. 127-133

Thompson, C. W. N. Procuring systems: Experiences from civilian

and defense agencies. Proceedings of the Seventh Annual Acquisition

Research Symposium , 1978b, pp. 329-333*

Thompson, C. W. N., and Rath, G. J. The administrative experiment:

A special case of field testing or evaluation. Human Factors , 1974, ]^(3), 238-252.

Urban, G. L. A model for the management of a family planning system.

Operations Research , 1974, ^(2), 205-233*

Vazsoni, A. Decision support systems: The new technology of

decisionmaking? Interfaces , 1978, ^(1), 72-77.

Waller, J. D., Kemp, D. M. , Scanlon, J. W. , Tolson, F., and Who ley,

J. S. Monitoring for government agencies (783-41). Washington,

D.C.: The Urban Institute, February 1976.
Welsch, D. R., and Lee, R. W. Adapting systems to cope with multiple factors. Management Focus , 1979 , ^( 1 ), 6 - 17 .

Youker, R. Literature review and selected annotated bibliography.

Project Management Quarterly , 1978 , ^( 2 ), 12 - 13 .

Zand, D. E. , and Sorensen, R. E. Theory of change and the effective use of management science. Administrative Science Quarterly , 1975 , ^( 4 ), 532 - 545 .

Zimmerman, P. J. Principles of design for information systems.

Journal of the American Society for Information Science , 1977 , ^( 4 ), 183 - 191 .
475

APPENDIX A

Background Materials on the Experimental

Technology Incentives Program
476

. ! f.xperime::tal techn’ology imce:;tivi:s ppqg?;a.m . |

ft (Background For A Briefing For Secretary of Comerce,

Elliot L. Richardson)

v.

HISTORY AND DEVELOPMENT

Background

The Experiutental Technology Incentives Program (ETI?) of the

National Bureau of Standards v/as created by Presidential

direction in early 1972.

( The President's decision was based on the studies of potential

means for stimulating the nation's technology which were

undertaken by a special team headed by Dr. Ezra Solomon of

the President's Council of Economic Advisers. This 'yas a part of the Domestic Council’s New Technology Study conducted

in the last half of 1971. Dr. Solomon's study group recommended

experiments be conducted to determine effective means for

stimulating industrial productivity.

The President's State of the Union Message of January 20, 1972,

stated, "Over the last several months, this administration

has undertaken a m.ajor review of both the problems and the

opportunities for American technology. Leading scientists

and researchers from our universities and from industry have

contributed to this study. One important conclusion we have

reached is t-hat much m.ore needs to be known about the process

of stimulating and applying research and development. In

som.e cases, for example, the barriers to orogress are financial.

T — T
477

2 .

In others they are technical. In still other instances, customs,

habits, laws, and regulations are the chief obstacles. We

need to learn more about all these considerations —and we intend to do so.... But while our knowledge in this field

is still modest, there are nevertheless a number of important

new steps which we can take at this time. I will soon present specific recommendations for such steps in a special

message to the Congress."

The President's Economic Report of January 1972 indicated

the importance which technology is perceived to have in the

American economy by stating "Investments in scientific

know’ledge and in its application to productive uses have

become an important characteristic of the American economy.

Benefits from the development and utilization of knov/ledge

are many and varied. They are evident in improved health

for millions of Americans as well as in our greater understanding of outer space. They include entirely new products

that enhance the quality of life and new techniques that

expand the productivity of the Nation's human and physical

resources. While an accurate evaluation of those benefits

that directly improve economic performance is difficult

to say nothing of the less tangible benefits —it is widely

agreed that the group of activities called research and

developm.ent (R&D) plays a central role in our economy. It has led to new products and industries; a.nd it can contribute in important ways to solving today's complex economic

and social oroblems.
478

3 .

The President's Science and Technology Message of early 1972

provided specifics about the N'BS Program. The Science and

Technology Message states the new Program was "to determ.ine

effective ways of stimulating non-Federal investm.ent in

research and development and of improving the application of

research and development results. The experiments to be set

up under this Program are designed to test a variety of

partnership arrangements am^ong the various levels of government, private firms and universities."

f Purpose

In his presentation to the Congress as a part of the FY73

budget process MBS Director Lewis Branscomb identified the new MBS Program as being a response to the "need to learn."

Branscom.b further indicated that the new Program's final measure of success v;ould be whether or not its results

survived in the m.arlcetplace.

Commerce Secretary Peterson indicated in testim.ony before

the Congress that he oerceived the new Program as part of

the Goverrmnent ' s encouragement of greater private sector P&D

by improving the climate for it.

Thus the purpose of ETIP, as envisioned by those responsible

for the conduct of the Program, can be summarised as the

carrying out of experiments to learn more about the orocess

of stimulating and applying technology.
479

4 .

On April 6, 1973, Commerce Secretary Fred Dent approved and

transmitted a Program Plan for ETIP to the Director of the

Office of Management and Budget. That plan stated "The

objective of the Program is to learn how the Federal Government can provide policies and incentives which will encourage

greater technological innovation in the private sector."

ETIP's refined Program Plan was approved by Commerce Secretary

Dent at the conclusion of a briefing on February 1, 1974,

and it reaffirmed that ETIP's purpose was "to initiate a series of experiments to find better ways to encourage

private investment in R&D,"

Approach

The approach to that knowledge-gaining objective has evolved

over time.

Initial thinking was that ETIP would, in cooperation with

industry, launch technology-stimulating projects . Secretary

Peterson provided guidance that he did not believe that the

Department of Commerce should pursue industry-specific

projects unless given a Congressional mandate.

A second approach considered was that ETIP would focus on

the effectiveness of various mechanisms such as (a) aggregation of industrial R&D capability to address industry-wide
480

5 . technxcal problems, (b) new methods for transfe ' held technology, and (c) market a g°vern:r,enttechnology This

^-^^egation to stimulate

gy. Thus approach was determin^H

because of the absence of

^ feasible

ence government enti^^oo u- react to these var. ^txtxes whxch could

e varxous mechanisms.

A tMrd

development - - of

P^ooeoeep , love.:! .I

~

-ovation ta.e pUoe within the InduPtmal oo- identify „a,3 3,

P-P/ to

^ abandoned because o-' the

' ^PProach was

^ Oj. the widely helH

complexities o^' the ' ^^^Ption that the

, ,

" ‘ rnnovatxon process precluded useful

Still another approach consxdered was the exam^naf S££i2na^prob_lem^^ ,, . , , --am.natxon of

- -ch new technology could be

the investigation of ways to -acM^ introduction of such -ech t ^ =-lrtate the

discarded '

' ^ '^hnologxes. Thxs apprcach was ed oecause national problem ar^as tend *- . attention of manv Federal

.Hthose

agencxes and the active ^^'es of

agencxes precluded the establishment o^* mea^^ - experiments. ' ‘^eanxngrul

Tbe approach

consideration

lated to how

and direction

d>cd.ectec tor the
481

6 . In the pursuit of this selected approach, it was determined

that it would not be practical to conduct experiments on a single firm basis because of the very large number of such

experiments that would have to be conducted before one had

sufficient data for determining • public policy. It seemed to

make good sense that the governmental agency responsible for implementing policy changes in a specific area should be involved in

ETIP's experimental activities. Governmental agency participation

in ETIP's activities would not only assure intense interest

in the conduct of the experiments, but also largely overcome

the problems of aggregation. A selection was made of an initial set of Government policies based on pragmatic

considerations which limited the number of experimental

opportunities which could be simultaneously pursued. These

consideration” include:

• The policy questions selected should be those

for which there is a reasonable expectation for

obtaining answers based on experience.

• The policy questions selected should draw on the

capabilities of the Departm.ent of Cominerce and

should be related to the broad mission of the

Departm.ent.

• The policy questions chosen should be those where

the private sector would be expected to respond.

Areas chosen for concentration are orocurem.ent policy,

regulatory policy, civilian research and development

policy, and economic assistance policy.
7 . (Areas considered for experimentation but not selected for

early attention include patent policy, trade policy,

tax policy and antitrust policy.)

General Hypotheses

The background statements give rise to two implicit general

hypotheses:

• There exist forms of government policies capable

of modifying the rate and direction of technological

change.

• An appropriately designed set of experiments can both test the hypothesis above and uncover "better

ways" for formulating and implementing those

policies

l?oal ETIP** s goal is the development of a set of policy guidelines

on technological change and the body of knowledge necessary

for their effective use. ETIP will, in the course of its

efforts, stimulate changes in technology. (Such changes

are a by-product of its primary task. The significant

outout of ETIP must be pracrmatic and effective

methodologies that the Federal Government can use in ^

stimulating the entire process of technological change

within the economy in general.)
483

3 . Operational Strategy

ETIP ' s Program Plan states that "ETIP must provide both for

the development of new policy guidelines and for their

adoption by those agencies that are to use them..."

ETIP experiments and studies are directed towards testing

and analyzing specific sub-hypotheses of the hypothesis

associated with each policy area. When possible sub-hypotheses

are drawn from the existing literature and prior research.

When prior research and literature are not adequate to

permit the formulation of sub-hypotheses, then ETIP conducts

background studies before initiation of experiments. ETIP

has commissioned a number of such background studies and

analyses to provide appropriate understanding.

ETIP has adopted an operating strategy of working in partnership with responsible mission agencies in the conduct of its

experiments. Because of the partner's greater experience in

the specific subject area of the experiment, that agency

normally takes the lead in the conduct of the experiment and

ETIP operates in a supportive posture. ETIP's contributions

to these experiments are the provision of a rigorous design

and the conduct of a careful evaluation with the expectation

that the experiment will yield unambiguous data which will

significantly contribute to the resolution of a policy

issue.
484

9 . It is recognized that the agency's interest is primarily in

the specifics of the individual experiment as it contributes

to the agency’s mission, while STIP's interest is primarily

focused on the process being used in the experiment.

Continuing interaction is needed in order to meet the interests

of both of the partners. The partner agency is asked to

commit significant resources ;.to the experiment. This

increases the leverage of ETIP's efforts. It also insures

the continuing interest of the partner agency.

£

This operating strategy means that the partner agency which

must implement the procedural change learns with ETIP during

the course of the experiment. The agency is then ready to

undertake change when warranted on its own initiative.

This strategy precludes the need to implement change by

high level directive. It rests rather on an incentive of

instituting change for the im.provement of agency performance.

Measures of Success

The most appropriate measure of the Program's success is

that asserted in its approved Program Plan. That measure

is "the number and quality of the ETIP policy guidelines

adopted and used." Although specific technical improvements

will occur in the conduct of many ETIP experiments, those

results are not measures of ETIP's progress toward its

objective. Specific technical changes do, however, indicate

that ETIP's policy guidelines can be used to influence the

rate and direction of technological change.
485

10 . SPECIFIC EXPERIMENTAL POLICY AREAS

Procurement

The hypothesis underlying ETIP’s procurement activity is

that the rate and direction of private sector innovation

can be influenced by using government purchasing power to

reduce market entry risks for innovative products.

In this area ETIP has initiated 25 projects with a total

ETIP funding commitment of $3,201,000.

In the procurement area, experiments are being conducted

in cooperation with the General Services Administration, the

Veterans Administration, and other agencies including State

and local governments to learn how changes in procurement

policies and practices can stimulate innovation by private

firms selling to government. The procedures used by the

Federal government can also be used by State and local units

of government. The efforts in the procurement subprogram

are intended to identify, investigate, test, and recomir.end

changes in procurement policy that will stimulate a market

for technologically improved goods. Activity in this area
486

11 . concentrates on experimenting with changed orocurement

practice through the use of such techniques as:

• Life cycle costing

• Value incentive clauses

• Performance specifications

• Warranty considerations

• Multiple award contracts

• New product introductory schedules

• Multiple year contracts

• Cost plus oontracting

In order to gain the oonfidence of the agencies who have

responsibility for proourement polioy, ETI? adopted a technique of experim.enting first with those procurement

techniques where there was high likelihood of provoking

response on the part of the private sector. Thus ETIP

concentrated on life cycle costing, value incentive clauses,

and the use of performance specifications. Progress in the

conduct of experiments with those three techniques has been

encouraging.

• GSA has estimated savings on the following bulk

orders of household appliances as a result of the

use of life cycle costing:

Air

Conditioners Water

Heaters Electric

Ranges

Cas Ranges

Refrigerators

Freerera

Energy reduction Net eavlr.ga (ure cycle) 2i: $423,000

13Z $326,000

7Z $121,000

7Z $1,545,000

15Z $377,000
487

12 .

• The Federal Supply Service now incorporates a

value incentive clause in all contracts for more

than $100,000 that involve specifications of some

type. The value incentive clause permits product

change suggestions by suppliers. Any savings to

the government are shared with the vendor. vrnen

the Government's cost of ovmership is reduced,

the contractor is awarded 20% of an average year's

savings as his share. Reductions in contract

i price during the term of the contract make the

vendor eligible for an incentive share of 50%

of the instant savings.

Vendors have exercised value incentive clauses in

eight contracts. One contract was for the provision

of household moving containers. The contractor's

product improvement will i.ncrease the number of

times the containers may be used by a factor of five.

It is estimated that this change will result i.n the

Government saving of about $7,500,000 i.n the ne.xt

ten years, on the first years purchase of 10,000

containers. Among the ot.her contracts were two that

recommended modification to the design of computer

air conditioning systems and eac.h is expected to save

the Government about $400,000 in energy costs duri.ng t.he

next ten years. These two contracts eac.h involve an
^88

13.

estimated purchase by the Government of 20 units

for each of two specific sites. The total Government buy of such air conditioners is estimated as 1,000 units annually.

• Experience in writing performance specifications

has shown that this type of specification is

feasible in such products as ranges, frost-free

refrigerators, and lawn mowers.

• ETIP's procurement activity has expanded from the

federal focus to include State and local procurements

as well. Seven items have been initially selected

for attention at the State level. These are: air

conditioners, reflective materials, storage batteries,

cut sized office paper, two-way radios, copy machine

toner, and synthetic oil. States which are participating in these procurements include: Kentucky,

North Carolina, Michigan, Illinois, Pennsylvania,

Washington, Texas, Kansas, Colorado, Florida, and

Virginia.

Prior to the initiation of the ETIP procurem.ent experiments,

the Government bought on a lowest initial cost basis. This

resulted in many cases in the Government buying an item which

was well below the level of available commercial products.

The ETIP experiments to date have provided convincing evidence
489

14 . that uhe experimental techniques can improve the quality of

the products that the Government buys. This improvement in

Government procurement has been cost- effective but it is

the end toward which the ETIP effort is directed. etip is

J convinced that as industry becomes convinced that the experimental techniques will continue to be used, industry will

'

respond to the reduced market entry risk incentive for new

products, we believe that prior to 1980, it will be possible

to support the procurem.ent area hypothesis that government

purchasing can stimulate product innovation.

Regulatory

The hypothesis underlying ETIP's reguletory activity is

that the rate and direction of private sector inryation

can be influenced by using government regulatory power to

reduce uncertainty and other regulatory constraints. It

is further hypothesised that such use of government regulatory power can be made wit.hout interferring with the main

purpose of the regulations.

In this area ETIP has initiated 10 projects with a total

ETIP funding commitment of $2,597,000.

The ETIP plan for experimental

data gathering in a number of areas have been identified by

significant. They include:

regulatory projects recuires

problem areas. All of those

regulatory reformers as beinc
490

• excessive regulation

• unfavorable cost/benefit ratios

• inflexible, unrealistic standards, rates or

enforcement schemes

• lag and delay in issuing standards, approving

rates, or certifying products

• inadequate, unfair and unscientific decisionmaking

processes

. uneven, unsure or unfair ' enforcement systems.

Examples of ETIP activity in the regulatory area are;

• The joint ETIP/Nuclear Regulatory Commission

project seeking to reduce lag time for the development

of nuclear power plant standards has been completea.

The three draft standards were developed. They

concern: (1) certification of nuclear materials

managers, (2) discharge of thermal effluent into

surface waters, and (3) guidelines for evaluating

combinations of natural and man-made hazards at

power reactor sites. It has become clear that the

time to develop draft consensus standards can be

reduced from the normal time of about two years to

a few months. However the ma^or lesson that ETIP

has learned from this experim.ent is that addressing

the problem that is widely held to be a regulatory

barrier may in fact not be strictly a regulatory

barrier. this case acceleration of tne aevelO;-me..
491

of the draft consensus standards has been achieved.

But its not clear yet that the stages foilowi.ng the

drafting stage will work faster than the old process.

Thus in addressing a problem in standards, ETIP

has learned to focus on the entire process in order

to accelerate the entire standards development rather

than focusing on a portion of the process.

• The joint Federal Rail .^dministration/ETIP

project seeking to determine the feasibility

of experimenting with shipping of perishables

in a way that would stimulate innovation. It

is widely believed that the rate regulation

structure has been responsible for the railroads

not investing in new technology. As a result

a rapidly declining portion of perishable freight

has been shipped by rail. This project is e.xanining alternatives which might reverse this tre.nd.

It is intended that the results of t.his project

will be used by a party-at-interest to petition

the Interstate Commerce Commission for rate revisions

or other appropriate remedies. Such parties-afinreres

include the railroads, shipper cooperatives, wholesale and retail distributors and growers.
492

17 .

A joint project with EPA seeking to examme

alternatives with respect to rigorous product

testing at manufacturer's expense whrch may be

pursued without increasing health risks.

Specifically this project is looki.ng at alternative certification procedures for pest control

systems. Included in such i.ucn alternative procedures

ere: (1) a variety of ways in which financing

could be provided (including direct payments,

loans, loan guarantees, or the establishment of

revolving fund), (2) greater use of Federal

laboratories in performing effectiveness testing,

(3) government pro'/ided insurance, and ( 4 ) e.xperirnental use permits.

joint project is being carried out wit.h the

Federal Power Commission that seeks to reduce

regulatory lag which is ' hypothesized as being an important determinant in t.he decisions of utilities

to innovate. The project is testi.ng and evaluati.ng

a series of new procedural tools in cooperation

with the public utility commissions in the states

of North Caroli.na, Ohio, .Montana, California, and

New Fork. The procedural tools include: , 1 , case

load management, (2) new method for computing test

years, (3) revised rate structure, (4) the developmen:

of comparative measures of operating efficiency and

r
493

18.

service quality, (5) new methods for utility

proposal evaluations, (6) development of, performance

benchmarks for regulated industry, and (7) analyses

of long range planning.

• A joint project with the Food and Drug Administration has been initiated to experiment with postmarketing surveillance of adverse drug effects.

This project will provide a new alternative—

policy decision based on experimental evidence.

This contrasts with the two existing policy

options of maintaining the status quo or embarking

on a major regulatory change with no real confidence

as to its effects.

It is ETIP's belief that these experiments will yield

unambiguous information which will contribute to an improved

regulatory decision process. Understanding of those circumstances where regulations can influence the rate and direction

of private sector technological change must await additional

experimental results.

ETIP has discovered that more important than its dollars is

its contribution of ideas about what can be determined in

experiments that are important in the regulatory area. T.he complexities of the regulatory process require heavy commitment of ETIP's staff time for the successful initiacion and

conduct of regulatory experiments.
494

Research and Develocmpn^

The hypothesis governing ETiP's activity in research and

development is that the nearly 59,000,000,000 of annual

Federal funoing ror civilian research and development can be »cre effectively directed towards use in new and improved

products. Civilian research and development is defined as Federally-funded research ot.her than defense and space. The

Federal Government is not the main purchaser of t.he technologreal products developed by civilian research and development.

in this area ETIP has initiated eight projects with a total

FTIP funding commit.ment of 51,976,000. 5475,000 has been

devoted to conducting research on fiber textile ^

- textile, ana apparel

flammability in response to a Congressional mandate contained

in the FY 1973 Senate Appropriations Committee Report.

Civilian research and development programs have been viewed

by many people as t.he Gover.nment trying to fi.nd oractre

nsos for t.he technology it had developed. This was viewed

as creati.ng a technology push program. ETIP • s view is t.hat the applied research portion of this program can be with

proper organization and management pursued in a oroblem

solving mode. The research would then te.nd to become more

of a technology null process because t.he problem definition

activity would have tc involve more deeply the ultimate user

of the problem solving products being developed.

T - -
20 .

The research and development area is investigating incentives

and practices that might be emnloyed by Federal agencies that

fund civilian research and development to facilitate the end

use of the research and development results. Experiments

are being conducted with Health, Education, and ^Telfare,

National Science Foundation, and private sector organizations.

It is a v/idely recognized consensus that expenditure of

Federal civilian research and develooment funds (where the

Federal government is not the user of the product—development

of better police radios, for example), has generally failed

to result in a flow of products into the marketplace.

Government funded civilian research and development must be more closely coupled to the production and marketing decisions

of firms if it is to influence the rate at which the private

sector innovates. This program is investigating previous

Federal activity to establish the factors that characterize

projects which have resulted in commercialization and use of

the R&D results. ETIP is also pursuing experiments in this

area.

ETIP projects in this area are as follows:

• A study on the effect of federal funding of civilian

research and development as an incentive for

technological change concluded that federallyfunded civilian research and development is not

sufficient to bring about technological change in

the private sector to any significant extent. The
21 .

496

study found that commercialization of federallyfunded R&D is nearly always accompanied by

company policy measures that cause or stimulate market

demand. Technical innovation is most often "pulled"

into the marketplace through appropriate incentives

rather than "pushed" by federally-funded R&D. Therefore,

policies for federally-funded applied civilian R&D

should be formulated in the larger context of the

complex process of innovation.

i

• A study on analysis of federally- funded demonstration

projects concluded that: diffusion depends on market

pull rather than technology push; demonstration projects

appear to be weak tools for tackling institutional and

organizational barriers to diffusion; large demonstration

projects with heavy federal founding are particularly

prone to difficulty; on site project managem.ent seem.s

to be generally effective; dissemination of information

from demonstration projects has not been a serious

problem; and demonstration projects should have a narrow scope for effective use.

The report suggests a number of guidelines for federal

agencies to follow in planning and conducting demonstration

projects

Both the Energy Research and Development Administration

and the Environmental Protection Agency have arranged

to be briefed on the detailed guidelines and have
497

22 .

indicated their intent to adopt the guidelines

in their decisions.

• The ETIP project seeking improved flammability of

cotton/polyester blends through the use of a consortium has led to the oreliminary finding that a consortium composed of individuals and institutions

representing the wide spectrum of interests present

in research and commercialization can be an effective

mechanism for directing government research towards -

practical marketplace use.

• Early results from the joint ETIP/NSF project seeking

to accelerate the flow of technology from universities

provides evidence that the number of invention disiir closures having commercial potential can be significantly increased when the ’university offers incentives

to the individual who has made the invention.

• The HEV7/ETIP effort seeking to cause the im.plemenpatuon

of the integrated utility system at universities provide

evidence that shows that the federal government can induce the use of technology, where it is cost effective

through the provision of engineering and economic

information. The implicatiUn of these findinas is that

under certain circumstances the governm.ent need not

mount expensive demonstration projects but can leverage

with rather modest amounts of funds the expenditures of

capital funds by the users of the technoloay.
498

Economic Assistance

The economic assistance policy area embraces ETIP's investigation of subsidy policies and also includes ETIP's activity with

small business assistance policies. (This expansion of interest

is based on ETIP's conclusion that the mechanisms for

providing financial assistance to small business are included

in the microeconomic policy tools applicable to the general

economy.) The main hypothesis underlying investigations in this

area is that Federal economic assistance programs can be used as

an effective tool for stimulating the use by the private sector

of desirable new technology. In addition, a number of these

programs, while not having technological change as an explicit

objective, nevertheless impact this important element of economic

growth. Unfortunately, t.hese impacts often take t.he form of

barriers to the development and adoption of new tec.hnologies . Thus,

policymakers must be sensitive to t.he technological i.mpacts i.n a wide variety of assistance programs.

In this area ETIP has initiated ten projects with a total ETIP

funding commitm.ent of $2,165,000 .

ETIP has selected as subjects for experiments:

• subsidize users rat.her than providers of a

good or service

• couple rate and direction of subsidies with rate

and direction of relevant tec.hnological change

so that the level and nature of the technology
499

24.

embodied in the capital is appropriate

for efficiently providing the output required to

meet the existing social demand

• couple operating/ capital/ and price subsidies

• do not constrain technological alternatives by

subsidizing only certain approaches to a problem

• improve demand forecasting as input to subsidy

planning and allocation

In this policy research area the results of two important

studies may be summarized as follows:'

• The first study involved analysis of internal

venture capital market imperfections. The

research on this question found no evidence of

substantial market imperfections that restrict

the flow of funds to small technology-based firms

and no indication that small technology-based

firms paid higher rates of interest or returned

their unaffiliated stockholders more than other

small firms. The study further found that there

was no evidence that suppliers of funds of small

technology-based firms earned higher profits than

could be earned by investing in listed securities

and no elements of the structure of behavior in

the venture capital industry which would enable

firms in the business to charge higher rates or earn greater profits than necessary to compe.nsate

for the risks assum.ed. However, a number of new securities regulations have been enacted in recent

years for whic.h impact data are .not available. Thus,

this area should be researched further.
500

The conclusion that there are no substantial

capital market imperfections does not necessarily

imply that the supply of funds to new technologybased firms in some sense is ideal. This is

because many external factors which influence

the investor's expected risk/reward ratio act

on this market. Thus, the study concluded that,

with a given risk/reward ratio, attempts to increase

the flow of funds by direct government investmient

in small technology-based firms may tend to displace private funds rather than increase the

total flow of funds.

• A study to develop an understanding of what federal

policies are needed under what circumstances to deal

with major economic disruptions in markets for

critical raw materials reached two general findings.

These findings are;

- Due to its flexibility, stockpiling is

generally an effective policy tool. Other

policy tools have lesser net potential benefits

because of the flexibility of stockpiling.

While this conclusion applies generally, it is

not applicable in every market studied (for

example, energy).

Technical knowledge about substitute materials

that might be used and other technical inform,ation that would be of value to users of raw materials is widely scattered. Information
501

26 . diffusion channels are inefficient, fragmentary,

and sometimes inaccurate. This situation means that consumers of raw materials cannot effectively

plan for potential sudden shortages. They also

cannot participate intelligently in the

development of technical alternatives.

ETIP may have a substantial experimental

role in diffusion of technical information.

t In addition to these general findings and many

specific ones in the individual commodity studies,

the contractor developed a reasonably general

computerized method for evaluating policy options

including supply and demand side technological

options, stockpiles, tariffs, and subsidies.

Three projects with small business focus have matured

sufficiently to provide meaningful results.

• A study seeking to evaluate alternative policies

that the government might pursue in assisting

small firms to comply with impacting government

regulations found fragmentaoion of regulation

affecting small firms . . . problems raising capital

for "non-productive" investments in compliance

techniques ... a tendency to seek "end of pipe"

rather than "process" solutions ... economies of

scale in complying with environmental standards . .

lack of small business voice in Washington.
502

The reT^o^t recor.\rr,GP.dGd thQt SBA intervene in the

legislative and regulatory process as an "active

but objective" advocate of snail business interests.

The report further recommended increased fi.nancial

assistance via loa.ns and lease guarantees, more

technical assistance and encouragerrient of various

types of cooperation a.mcng small firms.

9 The ETIP project with the Connecticut Product

Development Corporation has indicated the ma.ny administrative difficulties whic.h must be overcome in the transplantatio.n of an orga.nizat ion

mechanism from one political environ.ment to another.

It is reasonable to conclude t.hat the ability to

evaluate the technical merits of individual products is one of the least critical item.s in

determi.ning the viability of a quasi-state activity

such as CPDC. Rather, it is the economic or market

potential of the new product and the m.anagem.ent capability of the firm ’-/hich are the deciding factors.

9 The ETIP sponsored evaluaricn of the Small Eusi.ness

Administration innovation loan program, fou.nd that

the federal ta.xes ge.nerated by those projects

that v/ere successful exceeded the am.ou.nt of wrrte-off

of loans associated with those projects than were

not successful. The study supports t.he .hypothesis

that fi.nancral criteria can be relaxed when making

loans to firms that want to innovate wit.hout

suffering na^cr loss of funds due to default.
28 .

ETIP's experience in the Economic Assistance Policy Area

has shown that experimentation with economic assistance

policy is feasible. Such experiments are expected to provide

a better information base upon which to make policy decisions.

Evaluation

In this area ETIP has initiated eleven projects with a

total ETIP funding commitment of $2,068,000.

A major activity of the ETIP Program is the conduct of full and

careful evaluations with respect to experiments. These evaluations involve the selection, collection and analysis of

information to permit the assessment of the actual effects

of the experiment and its secondary impacts in the presence

of rival activities. Based on this assessment, the evaluation

activity will then seek to draw conclusions as to whether or not the benefits of the experimental intervention have been

a gain to society. The initiation of ETIP's evaluation

projects is following the same development as did ETIP's

experiments. In FY 1976 ETIP initiated the evaluation of

its procurement activity to be followed in FY 1977 by

evaluation of regulation and 1978 by evaluation in economic

assistance. Each of these three policy area evaluations

will be comprised of two separate efforts, one that evaluates

the impact that the experiment has had upon the agency

partner of ETIP and the other the impact that the experiment

has had in the marketplace.
504

29 .

In order to provide evaluations of experiments ahead of the

schedule outlined above, special evaluation activity for

selected projects has been undertaken in all policy areas to provide preliminary results to guide subsequent management

and direction of ongoing experiments. In procurement, field

surveys have gathered information from appliance manufacturers

in an effort to learn their reaction to the ETIP/FSS appliance

experiments. Funds have been transferred to the Nuclear

Regulatory Commission to evaluate the cost effectiveness

of the procedures that resulted in the acceleration of the

development of consensus nuclear standards. ETIP's right to

carefully observe the activities of the Connecticut Product

Development Corporation are being carried out by a contractor

who is collecting, summarizing, and interpreting data on the

Connecticut experience. Separate contract activities are providing real-time observations of the experiments in the

flammable fabrics and integrated utility system for the

purpose of providing definitive case histories of these two

activities. The evaluation of this kind will be absorbed

into the main evaluation systems being developed as they

come on line.

One important purpose of t-he evalution effort of ETI? is to

provide definitive information such that a decision can be

made in FY 1980 as to what the future of ETIP should be. It
505

30.

must be remembered that although every effort will be made

to have the necessary information developed for the 1980

decision, the phenomena being researched are contained in a

complex field situation.

ETIP's experience with the conduct of evaluations to date

has shown that its experiments can be evaluated. Businessmen have been very open in discussing with evaluators their

plans for technological change and the impact that ETIP '

work is having upon those plans.

GENERAL CONCLUSIONS

ETIP has tested the hypotheses listed in each of the specific

experimental areas. The level of current knowledge with

respect to those hypotheses are shown at the end of each

of the area discussions. With respect to rhe general

hypotheses, ETIP is convinced that each of the four policy

areas under investigation do m.odify the rate and direction

of technological change. ETIP is also convinced that the —

cooperative experimental approach is a viable means for

testing hypotheses and uncovering "better ways" for formulating and implementing those policies. However, at this

early time we do not have sufficient evidence to demionstrate

specific rates and directions of technological change as a result of the limited number of policy guidelines investigaced
506

31 .

In the course of the design and conduct of its projects,

ETIP has found that the complexities of policy analysis

require more work in the design of experiments than had

been anticipated. Specific documentation of the project

design in considerable detail is needed before the initiation

of the experiments. Close and continual interaction with

the other agency partner is a necessity to assure t.he maximum policy output from ETIP's activities.

The ETIP Program may be characterized as having survived

a difficult birth and grown through the learning process

of adolescence and is now just beginning to have the

\

maturity and 'understanding needed to develop the policy

recommendations v/hich are its reason for being.

September 15, 1976
C^^/Y) /r^

^/,r/7>

UNITED STATES '-'A

GENERAL ACCOUNTING OFFICE

V\ .

' ^ J *>i .

.. - 4-.'.' ,.i .

' " 'A'

'

t :’,t.C::-.

Potential Problems That Should

Be .Considered In Evaluating

Th

e' Experimentat Je c h n o I o gy

I ncen ti ves' Progra m

Department of; Commerce T . National ’’ ' Bureau/of Standards , » **d \ % • ' I ' 'i W • ' "•'•’>• -1 •« • J

' I 'FiNy-'''-''

.v, . - . .- v\' AA’fA-Aj A-'-i -

'

' Under^the Experimental Technology Incohlives Program studies and experiments are conducted ' to develop' policy recommendations orr how the Government can affect the

rale at which the private sector innovates.

GAO has identified problems in Program per- formance which could hinder the Progtarn's

ability to meet its objectives.

The Department's current evaluation of the

Program should determine the extent to which these problems have an impact on its performance. If it is found that the Program's

objectives are not being achieved, then action

should be taken to overcome them and/or

modify the objectives.

-* T;<

• # V* 'T’’-

'

^ -x- == >

- A

;> !* • A?

PSAD.77-132 JULY 25, 1977
508

United States Genei^al accounting office

WASHINGTON, O.C. 2C5A3

CMMUNtTY AND ECCf^MIC

CEV£1»CPM£NT CIVISJON

3-1U32T

The Honorable

The Secretary of Corrmerce

Dear Madam Secretary:

We have been conducting a survey of the Experimental

Technology Incentives Program of the National Bureau of Standards. '

In view of the evaluation of the Program recently started by your

Department, v/e are discontinuing our survey. However, the infcrmaiion gained and observations made during our survey show potential problem areas that could hinder the ability of the Program

to meet its objectives of (1) conducting coordinated studies and

exoeriments with Government agencies to test and evaluate how Gov- ernment policies affect the rate at which the private sector innovates, and (2) publishing definitive reports that evaluate the

results of the experiments and recommend appropriate policy.

We discussed these potential problems with the Assistant

Secretary for Science and Technology and other agency officials

concerned with the Program. We are bringing tnem to your attention for consideration in your Department's evaluation and forthcoming budget decisions for the Program.

Science and technology experts believe that 'the Program is directed toward an important need and parti ci oati ng agency of'icials

saic that their association with the Program had been beneficial.

However, Program personnel have encountered management difficulties

that have hampered Program activities and could continue to do so

in the future.

The size of the present staff and its reliance on othe" agencies to assist in conducting coooerative exoeriments have made it difficult for Program management to plan and conduct the number of

experiments considered necessary to obtain information to make policy

recommendations. The effectiveness of the method being used for

evaluating experiments has yet to be determined. There also seems

to be a need for more effective interaction between Program personnel

and Federal policy-making and policy researcn organizations having

similar or related objectives. Also, the Program should remain in the National Bureau of Standards unless the acvantages provided by

its location can be matcned or surpassed elsewne^e. These matters

are discussed more fully in tne aooendix.

1

i

I

I

I

I

I

i

I

I
..he Prosren. ^='?TM'"^''’^f„f"^° 1 c^^ecOTdndet1 Gn on »hat

on i-.s po-.ontiai pH»ata sec-.or innovation,

the Government can

„f the Procram are not being

If it is found that action should be taken

achieved because “f •->'“= SnJ’Si’oM-.tives.

to overcome them and/or modify the o Jae shall be pleased to discuss the

S.

i-e/raS^S: t!^n-vart°“o’l?ert“or!“pJoLh-^ent and Systenis Acpulsltlon

Division on Z75-3195.

7-!fi n- th“ L“cislative Reorganization

As you know, AeO;'d"

t^oeral agency to sutmit a wri.-

Act of 1S70 h^duines Vhe 1;=^ ; reconsoendations to the House

ten statement on actions .aKeno

senate Committee on

Cormiittee on Government Operations and ,,, ^ate of the

Governmental Afrairs

r'op^,it-=.=s on Aporcpriaticns with “

days after the date of the report.

We are sending copies or ice of Managefour committees

of'” the lenate Cotmerce. Science

m®nt and Budget; and tne^Ch • ,, interstate and Foreign

and Transportation -tee , .he Ko

. Technology Committee.

Comnerce Coimiittee, and the House tcienu,

Henry Eschwege

Director
510

ISSUES -03 CONSISER.J'TION IN EVALUATIiNG T-E

EXPERIMENTAL TECHNOLOGY INCENTIVES ^RGSPA*-*

ORIGIN OF PROGRAM

In the summer af 1971, the Nixon Admini stration attempted

to bolster the lagging economy. Adverse conditions included a high inflation rate, a worsening foreign trade balance, and a high

level of unemplo>'ment, including a large numoer of scientists and

engineers. With the growing realization by economists that research

and development has a positive impact on economic growth and productivity, Aami ni stration officials began to look for ways to apoly

scientific and technological resources to civilian sector problems.

In July 1971, the President launched the New Technological

Opportunities Program to generate proposals for new Federal initiatives in science and technology. This effort initially consisted of three elements: Office of Science and Technology personnel reviewed proposals for new technology orojects from Federal

agencies; a member of the Council of Economic Advisers heaced a study of potential economic incentives to spur the funding and

utilization of RSD in industry; and a grouo headed by the Treasury

Department dealt with the transfer of technology among nations.

Ooinions were solicited from industry executives and scecially

convened advisory panels. This entire effort was coordinated through

staff of the Domestic Council and ultimately reviewed by White

House staff.

The New Technological Opoortunities Program generated many

proposals, but by Decemoer 1971, the White House staff recomm.ended

that no major funding be provided. It was felt tnat there was

no sound basis for funding major new orojects because not enough

was known about how the Government could bring aoout change in technological innovation in the marketolace. As a consecuence

officials at the National Bureau of Standards and the National

Science Foundation were told to design experimental programs to test various means of providing this information. The new programs

were included in the Fiscal Year 1973 budget request.

On March 16, 1972, the President sent a science and technology

message to Congress whicn included goals for three new programs.

The Experimental Research and Development Incentives Program at

the National Science Foundation and the Experimental Technology

Incentives Program (ETIP) at tne Nat'onal Bureau of Standards were

conceived to "

. . .determi ne effective ways of stimulating non-Federal investment in research and development and (imorove) the apolication of researcn and cevelopment results." The experiments of

1
511

APPENDIX I APPENDIX I

the programs were to be "...designed to test a variety of partnership arrangements among the various levels of government, private firms and universities."

Also, a study program known as the P.esearch and Development

Assessment Program was established at the National Science Foundation to "...support assessments and studies focused specifically

on barriers to technological innovation and on the consequences of adopting alternative Federal policies which would reduce or eliminate these barriers."

Our work focused on the Experimental Technology Incentives

Program's management and activities. Our survey was performed

at the Department of Commerce headquarters , the National Bureau

of Standards, the National Science Foundation, the Office of Manage.ment and Budget, and several other Government agencies and groups

involved with the Program. We also obtained views on the Program

from acknowledged experts in science and technology policy. Our

observations follow.

THE ETI? PROCESS

The strategy of the Program is first to identify Government

functions that may influence whether and in what ways private industry makes use of new technology and then to develop experiments

in cooperation with other Government agencies and private sector

organizations, testing whether new policies or procedures can lead

to more affective use of technology in private industry.

Experiments and studies in three areas of Government policy

are being conducted: —procurement, including the testing of potential incentives

to technology innovation such as life cycle costing, value

incentive clauses, and performance specifications; —regulation, including changes such as alternatives to mancatory standards and reducing the time to establish

new standards; and

--economic assistance, including problems relating to venture

capital, small business policies, and cotocdities suoplies

and shortages.

Early projects proposed for the Program included policy subjects

such as patents, taxes, and anti-trust administration. Cooperation

witn industry to develop soecific tecnnologies was also consiaered.

However, these proposals were not accepted.

2
APPENDI I p?e?:dix I

In February 1974, the Secretary of Ccrmerce aporoved the present

ETI? Plan and released funds previously approoriated. As of March 1,

1977, there were 17 full-time staff positions and 78 projects had

begun, consisting of 25 experiments, 22 studies, 12 evaluation projects and 18 other projects. As of September 30, 1975, S14.7 million

had been ooligated and for Fiscal Year 1977 S3.1 million has been

apportioned.

The methodology of the Program for gaining information that

might lead to policy reccnorendaticns is:

--researching broad technology-economic related issues to identify areas for potential study and experimentation

—working with an agency to define specific subjects likely

to result in some oolicy changes; —designing an exoeriment or a background study which can

lead directly to exoerimentati on , or designing a general

study if experimentation is inappropriate; —conducting the experiment or study in coooeration with the

agency based upon a project plan; —evaluating the project to assess its impact on the agency

and the economy; and

--following up on the information gained from individual

projects and recommending appropriate policy.

Vie have identified potential weaknesses in the activities of

the Program as discussed in the following sections.

PLA'INING. CONDUCTING AND EVALUATING

THE PROGRAii'S STUDIES AND EXPERIMENTS

Planning Experiments

The ETIP staff has found that background literature does not exist in many areas of Program interest and background studies

must be performed before experiments can be designed to test how

the Government can increase technological innovation in private

industry. A satisfactory methoc of studying and analyzing program

areas and designing experiments has not yet been developed.

The Program management has concluded that the ETIP staff must

perform problem area identification rather than relying on contractors. In dealing with contractors, difficulties encountered included delays caused by the contracting process, tne costs o* the

contracts, and the length of time required for the contractors
513

APPENDIX I APPENDIX I

to learn about agency operations. In addition, naving contractors

perfom studies does not allow the ETIP staff to gain the direct

knowledge through research that would be valuable in managing the

experiments identified as a result of the studies.

The Program staff has found that the potential cooperating

agency must help in defining the reasons for and scope of experiments. The staff needs to make sure that the objectives of both

ETIP and the other agencies are met through the cooperative arrangement and must overcome administrative barriers encountered due to the limited experience most agencies have had in working in a cooperative arrangement. A methodology for combining the resources

of ETIP, potential cooperating agencies, and contractors is still

being developed.

Designing experiments has been unexpectedly difficult because

of the innerent complexities of making policy analyses and working

with other agencies. Also conducting and monitoring on-going projects has limited the staff time available for planning new projects. The Program's manage.ment does not believe tne existing staff

level is sufficient to perform background research needed to identify problems and design experiments.

The experience gained in developing on-going Program areas

should be considered in order to formulate time and staff requirements for identifying problem areas and designing experiments.

This information should imorove future planning by permitting a better balance between program objectives and program resources.

Conducting ETIP Experiments

The Program plan approved in 1974 stated that many experiments would be necessary to develop general conclusions upon which

policy recommendations would be based. In the procurement area,

the strategy is to test known incentives in as many environments

as possible. Several subjects have been identified for experiments

in the regulatory area. However, the Program staff has encountered

difficulties that have slowed the rate at which experiments can

be initiated.

The strategy of the Program recuires that the staff be heavily

engaged in all phases of the experimentation process. The staff

initially underestimated the time required to effectively plan,

conduct, and monitor experiments. The activities of the Program

are far more comoiex than originally envisionec, partly because

the agencies cooperating with ETIP have different mission objectives.
APP'NDIX I APPENDIX I

We believe that the Depart-nent ' s current evaluation of the Program

should determine: —whether the present rate of experimentation is adequate

to develop information useful for making policy recom- mendacions within an acceocable period of time; and

--the extent to which increases in ETIP staffing could

increase the rate of experimentation.

Evaluating Experiments

The importance of evaluating exoeriments to provide a sound

basis for policy recommendations was noced in E7I? plans. However,

the designs for some experiments started in Fiscal Years 197A and

1975 did not adequately consider evaluation needs and there were uncertainties in initial experimental results. In January 1976,

ETIP personnel stressed the need to incorporate evaluation designs

in project planning. Thus, recant ETIP experiments may permit more useful evaluations than earlier ones. However, most ETIP experiments begun since that time have not been completed. Therefore, it is not yet possible to determine the extent to which improved project designs will result in useful evaluations.

Evaluation difficulties are major problems in obtaining useful

information from exoeriments on how Government action can increase

technological innovation in private industry. Experience to data

indicates that the following factors may limit the information

that can be obtained through evaluation or increase the time and

cost of evaluation activities: —the difficulties in convincing coooerating agencies of

the importance of rigorous evaluation plans; —the complex problem areas addressed by ETIP experiments.

For instance, it is difficult to relate the effect of

specific regulation changes on technology innovation; —the need to complete many exoeriments before drawing

general conclusions about a problem area; —the cost of collecting information necessary to reacn conclusions from experiments and studies. For example,

about S3 million has been obligated for ETIP procurement

experiments and studies and 31.3 million for evaluation of

their results. The ETIP staff estimates that evaluation

requires at least one half the staff required for conducting and monitoring projects;
515

515

APPENDIX I APPENDIX I —the length of time needed for oolicies to have an effect

on technological innovation by industry. The National Bureau of Standards estimates that it could take years

for new procurement experiments to produce technological

innovation in industry.

ETIP plans call for contractors to assess the impacts of ex- periments and studies in procurement, regulatory, and economic

assistance policy areas. In 1976, ETIP management contracted with

Stanford Research Institute and Research Triangle Institute to evaluate the procurement experiments and studies. The contractors

are to submit their evaluation plans to ETIP in September 1977

and evaluation reports in 1978. A contractor evaluation of the

regulatory area is scheduled to begin in Fiscal Year 1977. The

economic assistance evaluation is scheduled to begin in 1973.

It will not be possible to judge the effectiveness of the

present strategy of using contractors to assess the impact of ex- periments and studies until reports are prepared on the procurement

area in 1973. Thus, ETIP management does not now have a proven

method of gaining useful information from its exoeriments for making recommendations on technology policy.

INTERACTION WITH OTHER POLICY-MAKING

AND POLICY RESEARCH GROUPS

Aoenci es

The strategy of the Program is to conduct experiments with other

Government agencies. The staff believes that, by cooperating in ETIP experiments. Federal agencies can become familiar with the

use of new procedures and prepare for expanded use of procedures

being tested.

Of''^icials of cooperating agencies whom we contacted told us that the exoeriments have benefited their agencies. For example,

a Federal Supply Service official told us that the Program has

improved the Service's ability to develop new procurement procedures. The role of the Program in Federal Supply Service efforcs

to apoly life-cycle costing was discussed in a previous General

Accounting Office Reoort.l_/ Officials of other agencies told us that association with ETIP helped them become more aware of their

activities' effect on industry and that the exoertise obtained

from the Program staff helped them develop beater projects.

VLetter Report to the Administrator of General Services, Gene'*al Services Administration, PSAD-75-150, July 23, 1975.
APPENDIX P PEND IX

However, we are concerned about reliance of che Program on agencies to conduct cooperative experiments on those issues that

have teen identified as necessary to provide a knowledge base in

a policy area. For exa-mple. Program officials believe that multiyear contracting may encourage technology innovation in private

industry and information on this aspect of the prccuremient process

is necessary. However, they have not been able to reach an agreement with another Federal agency to test multi-year contracting.

Consideration should be given to the extent to which the inability

to reach agreements wi t.n other Federal agencies has hindered the

effectiveness of the Program.

Policy Groups

Since HTTP began, policy-mechanisms have been legislatively

established that might assist Program officials in reaching agreemencs with agencies to conduct experiments; —The Office of Science and Technology Policy, established

in 1975, is resoonsible for recotnnendi ng Federal policies

designed to advance the applicaiion of sciencific and

tecnnological capabilicies to national needs; —The National Cancer for Productivity and Quality of Working

Life, established in 1975, is responsible for developing a national policy for productivity growth in consultation with

the appropriate organisations in Governmenc; and

--The Office of Federal Procurement Policy, established in 197A, is responsible for providing overall direction of

procurement policy.

The oarticipation of these groups in identifying issues appropriate for experiments should be considered as a way to increase

the ability of ETIP personnel to conduct experiments on tecnnology

policy cuestions. Their involvement in the planning of ETIP experi ments could also enhance the usefulness of ETIP results in policymaking by these grouos.

ETIP staff members have coordinated informally with these

policy-making groups. For examole, the Program staff has keot

the Office of Federal Procurement Policy aware of its procurem.ent

exoeriments. As a result of a request by the ETIP Director, the Of

fice of Federal Procurement Policy issued a letter in January 1977

encouraging greater use of the Life-Cycle Costing technique in pro- curement. However, as noted in a previous GAO report !_/ , greater

1 /Manuf acturi ng Technology —A Changing Challenge to Imoroved Pro- ductivity, LCD-75--35, June 3, 197'.
17

APPENDIX I APPENDIX I

coordination among groups concerned with productivity and technology

could yield more effective results. Therefore, the Deoart-ment of

Commerce's evaluation of ETIP should consider the benefits of better

interaction with appropriate policy-making groups.

Improved coordination with Federal organizations conducting

research in areas related to ETIP experiments may also be beneficial. Personnel in the Division of Policy Research and Analysis in the National Science Foundation identify issues relating research

and development and technology innovation to national purposes,

and analyze policy options and their potential effect. The Division

staff has studied the effect of Government regulations on technological innovation and sponsored research on factors affecting industrial

productivity.

In the past, informal interaction has taken place between

personnel at ETIP and the Division of Policy Research and Analysis

on the planning and management of soecific projects. However,

as previously noted, the ETIP staff has concluded that it is necessary in many cases for them to perform background research before

initiating experiments to find out how the Government can increase

technological innovation in the private sector. Better interaction

may enable ETIP personnel to make more use of research perform, ed by

others in planning its experiments.

ORGAN IZATIGNAL LOCATION OF ETIP

The Congress has questioned whether ETIP should continue to be a part of the National Bureau of Standards.

In considering the institutional setting of the Program, we noted

that the Department of Commerce's mission includes stimulating and

supporting industrial research and development. The National Bureau

of Standards has a tradition of cooperative interaction with mission-oriented Government agencies and private industry. Also, the

Bureau's management understands the long time frames reouired for

experimentation and is sensitive to the uncertainty of the research

process

The location of ETIP at the Bureau appears suitable because

of the following characteristics of the Program;

--Program experiments must continue for 3 to 5 years before

results can be observed, —Access to other agencies is necessary for experimenting

and

—All experiments contain the risk of failure.

nO
APPENDIX I appendix I

^-n ha rr?"-n“d or surnassed else-where,

unless .hese advantages ^tp'^r^ss imoortana to success,

we te'neve tnat tne loca.ion - ‘o-iar-^d throuch adequate s.a.>-

than the ccn^i“ent or rr,anage!r,en ^ “

i nc anc runding suopocv..

pnNC'JJSIONS

* rTin has encountered diff’culIn nanaginq

ti »s and could continue to do -Jes thit hav. reliance

on ISendas^wlsaist in eiplrloients.

^el"5 -ad for evaluadln, axleri^AAs has yet to be deteralned.

is needed between

We believe that more

‘'I" policy research

ce-.cnnel at ETIP and Sreater

organitations tHat^ ave s.m la^^^

coordination and cooperation of necessary

objectives of the Program by -^e staff to conduct

bacKcround research, abe.trng - and enhancing the useexo-riments on technology policy ‘ ’4kars

fulness of experiment results to pol ..y-ak-rs

p rr.DMMENDATIQN

We >-eccinr.end that the problems imoact

tion of "S^cram -o'ceveloo policy

on the potenuial o. "-P® ' -- orate onvate secuor ,nnoon what the Governm,enu can do^.o

program are _not

vation. If it is round

-nrobl'^s, then the Secretary or bPing achieved bdcausg o, vhese ,pv,>y gpe obiecbives

Commerce snould act v-O cvV^r-jme

as necessary

the
519

APPENDIX B

A Manual for Designing and Implementing

A Process to Monitor

Complex System Developments
520

A Manual for Designing and Implementing

A Process to Monitor Complex System Developments

Preliminary Draft

Stephen D. Garrity

Washington, D.C.

May 1981
521

Contents

I. Introduction 522

1.1 How to Use This Manual 522

1.2 Who Should Use This Manual 523

II. The Monitoring Problem 524

II. 1 Complex System Developments 524

11. 2 Monitoring a Development 524

11. 3 A Research Role for Monitoring 525

11. 4 The Proposed Approach 525

III. The Monitoring Factors 526

III1 Introduction 526

111. 2 Design Characteristics (Factors 1-12 ) 533

111. 2.1 Operation and Performance Factors 533

111. 2. 2 Boundary Factors 535

111. 2. 3 Adaptation Factors 537

111. 3 Process Characteristics ( Factors 13-24 ) 541

111. 3.1 Introduction 541

III. 3*2 Task Structure Factors 541

111. 3. 3 Team Personnel Factors 543

111. 3. 4 Project Control Factors 546

111. 3. 5 Interaction with the User Environment Factors ... 55

111. 4 User Commitment Characteristics ( Factors 25-30 ) 559

III. 4.1 Introduction 559

111. 4.2 System Use Factors 559

111. 4. 3 System Support Factors 563

IV. The Monitoring Functions 569

IV. 1 Function #1 — Problem Identification 570

IV. 2 Function #2 — Strategy Development 570

IV. 3 Function #3 — Research 571

IV. 4 Function #4 — Documentation 572

IV. 5 Function #5 — Dissemination 574

V. A Model for Implementing the Monitoring Process 576

V.1 Introduction 576

V.2 Implementation Procedures 576

VI. Conclusion 584
I. Introduction

This document is the preliminary version of a manual to help

system developers design a process to monitor the development of com- plex systems. It is specifically aimed at projects where developers

are faced with substantial uncertainties over design requirements,

development processes, and the ultimate home for a system.

The manual is also expected to be of use to researchers of system

development processes. 3y combining research objectives with administrative use of the monitoring process, researchers can conduce structured investigations of system developments in real-time.

The manual is divided into several sections. First, the monitoring problem is identified so that readers can recognize how the

proposed process fits into a development. Second, a framework of

factors which should be monitored is described. Third, the functions

monitoring can serve in a development are discussed. Finally, a general model is provided on how the process can be implemented in a project.

Readers interested in examining the background behind development

of the monitoring process should consult the following document:

Garrity, S. D. Monitoring system development:

A framework and application (Doctoral dissertation,

Northwestern University, 1981).

I. 1 How to Use This Manual

Developers or researchers interested

tcring process may find it useful to scan

in using the proposed monithe manual before begi.nning
523

to read the details. It is suggested that all readers review the

Introduction section first in order to understand the general problem

under discussion and the basic components of the proposed approach.

After this, several different readings might be useful.

System developers may want to scan the section on monitoring

functions first to determine whether monitoring can enhance their current management practices. They then might turn to the section on monitoring factors to examine what factors they need for monitoring.

System development researchers may want to scan the monitoring

factors first to determine whether use of the factors matches their

current research interests. They then might turn to the sections

on functions and implementation in order to develop ideas on how to

conduct their research within an actual project.

1.2 Who Should Use This Manual

^

The manual has been developed from a government program where

information systems were needed. Thus, to some extent, the monitoring

process is oriented toward information systems rather than systems

in general.

However, researchers and developers involved with a variety of

complex system developments should find the manual useful for their

situations.
524

II. The Monitoring Problem

II. 1 Complex System Developments

Projects aimed at developing new systems where there are substantial uncertainties as to system requirements, development processes,

and ultimate ownership can present managers with a range of complex,

unstructured problems. General models or paradigms to guide system

development are of limited use in these cases since circumstances

are unpredictable and changing. A process to control development

activities as a project evolves is thus a key component for successful

management of a complex development.

II. 2 Monitoring a Development

A critical need in controlling a complex development is information. Developers must be able to sense when things are going well

or poorly and then be able to revise strategy accordingly. For ex- ample, if system requirements are initially uncertain, then it would

be important to ensure that requirements are established and subsequently reflected in a changing development process. Project managers

need to determine whether this is occurring in a satisfactory manner

and leading to an acceptable design.

The range of important factors involved and the need for timely

action suggests that information be gathered in a structured, routine

process over the duration of a project. Managers could use such a process to monitor key areas where problems may arise and affect development strategy. The process would provide a early warning of

situations requiring management intervention as well as establish

a base of information on which to analyze and implement change.
525

II. 3 A Research Role for Monitoring

A monitoring process can also be an important device for researchers of system development processes. Much of the current research

in the literature is based on reports of experiences or retrospective

analyses in which the researcher is either isolated from the project

or unable to implement rigorous designs. Systems researchers have

a critical need for more access to actual projects, especially in

early stages where the research can be implemented concurrently with

the project.

A monitoring process may provide the means by which researchers

can gain access to projects. An on-going monitoring system aimed

at administrative needs might easily be structured to also satisfy

research needs.

II. 4 The Proposed Approach

A monitoring process has thus been developed and is proposed

as a device which can meet management and researcher needs in a system

development process. The process consists of three components:

• A framework of thirty factors which should be monitored.

• A set of five functions which monitoring information can

be used for.

• A general model of how the monitoring process should be

implemented within a project.

A section is provided on each of these in the manual.
526

III. The Monitoring Factors

III.1 Introduction

This section describes a framework of factors to be used for

monitoring a system development. Thirty factors are identified and

they are divided into three categories: design characteristics, process characteristics, and user commitment characteristics.

Design characteristics are factors which describe what a system

looks like and how it works. Monitoring these characteristics is

important for several reasons. First, the design must be sound, meaning that it actually provides the functions needed by users as well as meet the specific performance requirements. Second, the design must

be attractive to users in order to gain their support and use. This

means that the design must help users solve important problems in a manner which is matched to their style and other organizational processes . Process characteristics are factors which describe the methods,

structures, procedures, resources, and personnel used to conduct and

control the development. Monitoring these characteristics provides

developers with an indication of whether process they are using

is producing the appropriate design and leading to full implementation

of the system at the end of the project.

The third framework category, user commitment, consists of factors

which characterize behaviors related to use and support for the system

in the prospective owner organization. Commitment is viewed here as developing over time, making it essential to monitor it during the

project and determine whether it is increasing or decreasing. The

final user decision to fully implement a system will be a reflection

of the commitment already established during development.

Two figures are provided to summarize the factors of the framework.

First, Figure III.1 lists the framework categories and all of the 30

factors. Second, Figure III. 2 provides a brief description of each

factor. These descriptions are also used in the more detailed discussions of the factors.
527

FIGURE

Til.

THE

MONITORING

FRAMEWORK
528

DESIGN CHARACTERISTICS

Operation and Performance Factors

General Description:

Factors

The functioning of the system design

in the user organization.

1. Response Time — The amount of time it takes a system to

respond to a user inquiry.

2. Quality — The accuracy, credibility, and utility of input

and output information in the system.

3. Cost of Operation — The amount of resources needed to operate

the system.

4. Input/Output Operations — The mechanics of user interaction

with the system.

5. Interconnection of Subsystems — The interrelationships of

system elements.

Boundary Factors

General Description: The borders of system and subsystem designs.

Factors

6. Capabilities/Limitations/Expectations — The conceptual boundaries of a system prescribed by the various groups of people

involved in the development.

7. User Groups and Their Interrelationships — The roles and costs

and benefits of involvement with the system pertaining to the

different groups of the user organization created by the system

design.

8. Interfaces with Other Systems and Organizations — The relationships between the system and other systems and organizations.

Adaptation Factors

General Description: The match between the system design and the user organization.

Factors

9.

Flexible Specifications — The match between design specifications and the existing uncertainties over system objectives,

processes ownership, products, etc.

FIGURE III. 2 BRIEF DESCRIPTIONS OF THE MONITORING FACTORS
529

10. Matching the System to the User — The match between the system

design and the user organization structure (structural), the

abilities, methods, and personal styles of individual users

(technical), and the personal relationships of individuals

in the user organization (personnel).

11. Novelty of the Design — The change a new system represents

over the existing system.

12. Evaluation and Updating — The provision of evaluation and

updating functions in the system design.

FIGURE III. 2 (continued)
530

PROCESS CHARACTERISTICS

Task Structure Factors

General Description: The design of project tasks to develop

the system.

Factors

13- Task Size — The amount of resources (money, manpower, and

time) assigned and consumed in the performance of individual

system development tasks.

14.

Task Priorities — The relative emphasis across tasks at a given point in time and the ordering of tasks over time.

Team Personnel Factors

General Description: The availability and functioning of project

personnel.

Factors

15. Skills — The availability of technical and interpersonal skills

in project teams as required by system development tasks.

16. Turnover — The change of personnel involved with project teams.

17. Commitment — The team member support for and implementation

of the goals, strategies, and tactics of the system development.

Project Control Factors

General Description: The structures and processes used to

control project activities.

Factors

18 . Crganization and Responsibilities — The structure of project

teams and responsibilities in the system development.

19. Decision Points and Milestones — The structure of specific

events created by project managers to recognize or review progress and decide future courses of action.

20. Reports and Reviews — The structure of written and oral commu- nication mechanisms used to document and review development

progress.

Interaction with the User Environment Factors

General Description: The involvement and contact between the

project and members of the user organization.

FIGURE III. 2 (continued)
Factors

21. User Involvement — The participation of members of the user organization in the managerial and technical teams of the project.

22. Problem Identification — The overall amount of attention to

and the amount of contact with the user organization in the

definition of user problems.

23. Testing — The amount of concept and design testing conducted

in the user environment.

24. Transfer — The amount of attention devoted to the transfer

of the system to the user environment.

FIGURE III.

(continued)
532

USER COMMITMENT CHARACTERISTICS

System Use Factors

General Description: User actions related to application of the

system in organizational processes.

Factors:

25. Applications — The use of specific elements of the system

in specific units of work of the user organization.

26. Consequent Actions — The actions resulting from areas of

direct system use.

27. Extent of Use — The amount of use relative to the number of

potential applications.

System Support Factors

General Description: User actions related to supporting applica

tion of the system to organizational processes .

Factors

28. System Champions — The emergence of advocates for the system

in the user organization.

29. Resource Commitments — The type and extent of resources

allocated by the user organization to support the system.

30. Changes in the User Organization — The alteration of policies

and procedures in the user organization in order to support

system operation.

FIGURE III.

(continued)
III. 2 Design Characteristics (Factors 1-12)

The 12 design factors are separated into three major categories:

(1) operation and performance factors, (2) boundary factors, and (3)

adaptation factors.

III. 2.1 Operation and Performance Factors

This category of design characteristics measures how the system

operates and performs tasks related to meeting user requests.

These characteristics have an impact on people using the system and

thus are ultimately linked to the success or failure of the system.

Monitoring them during development may decrease the likelihood that

inappropriate designs or poor performance stay uncorrected and act

as barriers to obtaining full support of the system.

Factor 1 — Response Time

Description . The amount of time it takes a system to respond

to a user inquiry.

Discussion . The time it takes a system to respond to a user,

usually in relation to a request, is an obvious design characteristic

which will affect use. Long delayed responses might be expected

to gradually discourage a user from interacting with the system.

Quick turnaround might accelerate introduction of the system and

build user confidence. Developers need to monitor the derivation

of the response time requirement and its development within the system.

Factor 2 — Quality

Description . The accuracy, credibility, and utility of input

and output information in the system.

Discussion . Quality is used as a measure of input and output

information characteristics, especially as to their accuracy, credibilit

and utility. This factor is important from two perspectives. First,

the system must provide correct information. Second, the system
534

must produce what is needed by users and lead to their support and

commitment to institutionalization. While these attributes can be

related, developers must ensure that the system has both since one does not necessarily follow the other.

Factor 3 — Cost of Operation

Description . The amount of resources needed to operate the

system.

Discussion . The cost of running a system is another factor which

developers should monitor. If a system is going to cost to much

to operate relative to its benefits or user resource constraints,

users will be less likely to support it. Obviously, there is a tradeoff

between the quality represented in the system design and the economy

in operating costs.

It is difficult to know operating costs in advance however,

especially if there are great technical uncertainties or little historical data on other systems which might be used as a benchmark. Developers

must thus closely monitor costs as a project proceeds and ascertain

whether the expected costs of operation are remaining feasible with

users.

Factor 4 — Input/Output Operations

Description . The mechanics of user interaction with the system.

Discussion . The operation of input and output functions of

the system is important to monitor for several reasons. First,

it must be mechanically easy for users to interact with the system.

If it appears too difficult for them to get a request in or out of

the system, most likely they will not use it or will find ways to

go around it. Second, the same conditions apply to personnel responsible

for providing data to a system. It may be difficult to collect the

data or there may be uncertainty as to how to put it into the system.

These problems could have significant impacts on staff support.

Generally, the design of input and output functions may conflict
535

with or enhance the way an organization works or solves problems.

Developers should spend time monitoring this characteristic.

Factor 5 — Interconnection of Subsystems

Description . The interrelationships of system elements.

Discussion . An important characteristicc to monitor is whether

and how elements are connected to each other. Developers should

consider organizing subsystems such that interactions between them

are minimized. Besides allowing for easier removal and update of

components, minimizing interactions may also help minimize the impact

of changes to the user organization or personnel relevant to these

subsystems.

III. 2. 2 Boundary Factors

A system can be described by what is included and what is excluded.

Similar descriptions can be used at the subsystem level. The boundary

of a system is an important concept to developers and its placement

can be very influential on development activities.

The main concern in this section is the conceptual boundaries

which define the system and the implications they have for the user

organization. Conceptual boundaries are the words, thoughts, plans,

objectives, expectations, models or roles represented in or guiding

design characteristics.

Factor 6 — Capabilities/Limitations/Expectations

Description . The conceptual boundaries of a system prescribed

by the various groups of people involved in the

development.

Discussion . Developers need to monitor how well design capabilities, limitations, and expectations match across user groups (and between users and developers). The system's anticipated capabilities

should not be oversold; unmet expectations may cause support to whither

away or be unattainable. Conversely, user expectations which are
536

lower than the capabilities claimed by developers may cause the system

to appear too grandiose and consequently gain little attention among

users.

Factor 7 — User Groups and Their Interrelationships

Description . The roles and costs and benefits of involvement

with the system pertaining to the different groups

of the user organization created by the system design.

Discussion . A system usually creates a number of different

groups in the user organization. For example, there can be system

owners, input sources, output users, system managers, performance

evaluators, or system operators. These groups may cut across already

established organizational lines.

Another way to identify system boundaries then is to study

these groups and their interrelationships. Each group has a specific

role in the system and certain costs and benefits associated with

it. Some groups may even have multiple roles, such as when owners

and users are the same people. Developers need to address both the

imbalances a design creates within a group (e.g., between the cost

and payoff of participating) and any resulting implications for

group interactions (e.g., cost/payoff differences between groups).

Factor 8 — Interfaces with Other Systems and Organizations

Description . The relationships between the system and other

systems and organizations.

Discussion . Various other systems or organizations can interact

with a given system. For example, they may be:

• Sources of data,

• Controllers of system resources,

• Users of system outputs, or

• Competitors.

Developers need to monitor the relationships to these outside

operations such that the proper ties are made in the design.
537

III. 2. 3 Adaptation Factors

Developers need to monitor how well the system and the user organization are fitting together. A poor match should be identified

early so that the design and/or organization can be adjusted. In

addition, developers also need to integrate the procedures to examine

the match into the final design so that they can be used to continually

improve the system during its operational phase.

Factor 9 — Flexible Specifications

Description . The match between design specifications and the

existing uncertainties over system objectives,

processes, ownership, products, etc.

Discussion . A system design should be kept flexible during

a development in order to accommodate (as appropriate) the uncertainties which are likely to arise. Uncertainties may exist from a lack

of knowledge about an area (such as a novel technical problem) or might evolve from changes in personnel, organization, or budgets.

In order to keep design specifications flexible, developers

should identify and monitor areas that have major implications for

the design if they in some way change.

Factor 10 — Matching the System to the User

Description . The match between the system design and the user organization

Discussion . As designs are created and detailed, developers

need to test the match between the design and the user. This will

help determine whether the design and the development strategy are satisfactory.

Three categories of matching seem to capture the major overall

concerns

1. Structural matching.

This is the match between the structure of the system

and the structure of the user organization. Structural

characteristics include goals, priorities, patterns of

communication, operations, relationships and
538

responsibilities.

2. Technical matching.

This is the match between the system design and the

abilities, methods, and personal styles of individual

users.

3. Personnel matching.

This is the match between the people in the user organization according to their relationship within the system.

Basically, the question is whether people will be able to

work together as required in the design.

Factor 11 — Novelty of the Design

Description . The change a new system represents over the existing

system.

Discussion . Developers need to monitor the degree of change

a new system represents relative to the existing system. It should

be expected that novel changes will be difficult to make or will

generate user resistence; incremental change may facilitate conversion

to the new system.

Developers need to monitor changes in the following areas

and determine implications for development strategy:

• System functions.

Functions previously unavailable may represent a significant

change to users. These functions may have been impossible to

develop before, or maybe they were not considered. There may

have been no preexisting need.

• Performance.

Dramatic changes in performance over an older system may

represent a source of significant change to users. While the

actual performance improvement could represent a small design

alteration, the implications of it may be much broader for all

associated activities.

• Operation.

New ways of doing things may be a significant change for users familiar with the older system.

• Other systems.

The relationship to other systems may change with the implementation of a new system.
• Support.

The organizational support for a new system might be very

different than that given to the old one. A new system may

create previously nonexistent barriers to its operation.

Factor 12 — Evaluation and Updating

Description . The provision of evaluation and updating functions

in the system design.

Discussion . In order to maintain the adaptability of a system,

developers need to ensure that updating functions are built into

a system design. Several different areas need to be examined

concerning evaluation and revision processes, including:

• Evaluation criteria.

Assuming that some kind of evaluation is needed, developers

should identify the decisionmakers involved and their criteria.

These decisionmakers might include groups well outside of the immediate user units. These criteria may be uncertain in early

project stages and may need development. Developers should think

through how these criteria can be applied and built into a design.

• Decision points.

If explicit decision points are used to control a project at ^

each stage (see also Decision Points and Milestones below),

developers should examine early what information might be needed

about the system. Implications from analyses like these may

possibly provide the basic strategy for later system testing

(see also Testing, Factor 23).

• Other organizational review processes.

If the system is to become an integral part of an organization,

its evaluation and updating functions may need to be tied to

already existing organizational review procedures. These might

include reviews of the organization conducted from the outside

as well. Developers may have to adjust designs to accommodate

changes from these sources.

• Freezing a design.

At some point it may be advantageous to freeze a design in

order to bring the full system into operation without continuing

distractions for revisions. Developers need to examine where

updates should be made and which ones can be delayed without

risking problems. In cases where updates are put off, provisions

should be made for users to initiate them when they assume complete

responsibility for maintenance of the system.
540

• Mechanisms.

Finally, developers need to examine how evaluation and updating

function overall. They should study whether problems are identified, whether redesigning occurs in a timely manner, and

whether changes actually are, or can be, made. This mechanism

must be operational like any other system component when the

system is finally institutionalized.
541

III. 3 Process Characteristics (Factors 13 - 24)

111.3*1 Introduction

A system development can be described by the methods, procedures,

personnel, resources, and structure developers use to operate the

project. These factors contain a process perspective in that they

are a means towards an end; the eventual design and institutionalization of the system. Factors which denote project activities like these

are defined as process characteristics in the monitoring framework.

The following sections identify and describe 12 factors. These

have been separated into four categories: task structure, team personnel, project control, and interaction with the user environment.

A special note should be made here concerning the different

groups which may be involved in the development. A development may

be the responsibility of several different institutions which are normally independent of each other. For example, contract support

may be needed or a third party may provide financial support.

The development process under multiple group arrangements like

these can be complicated. Responsibilities are shared, the different

groups must be coordinated, and different perspectives on strategy

may arise.

For this reason, some of the following sections for each factor

include discussion on how multiple group situations can affect a project, particularly in contracting cases. These discussions illustrate the additional characteristics developers should monitor

when more than one group is involved.

III. 3*2 Task Structure Factors

A system development can be divided into a series of specific

tasks or group of tasks. Tasks are usually grouped into categories

or phases, such as problem identification, design, or evaluation.

These demarcations are fairly arbitrary, however, and disguise the

true cyclic process and sequencing of development tasks.

Of more interest are the actual tasks which occur within and
across phases like these. Identification and coordination of these

tasks represent a significant problem to developers. For example,

tasks related to different subsystems may be similar and need coordina

tion. In other parts of the design, some subsystems may need to be

fully developed before others.

Developers need to monitor both the size and priority of tasks

as the development proceeds.

Factor 13 — Task Size

Description . The amount of resources (money, manpower, and

time) assigned and consumed in the performance

of individual system development tasks.

Discussion . The size of project tasks, or group of tasks, is one characteristic developers should monitor throughout a project.

The size of a task includes the amount of money, manpower, and time

consumed in performance of the task.

One of the main reasons to monitor size characteristics is to

control costs. Costs can easily expand beyond initial expectations

if there are major technical uncertainties in the design or numerous

changes of course in the project. Early and progressive monitoring

of resource expenditures may help avoid becoming overcommitted to

a narrow portion of the project or running out of resources at key

points.

While there is no one right size for project tasks, many sources

recommend dividing projects up into small, manageable steps which

are easier to control. There are several reasons behind this strategy

• Overall, system developments are complex undertakings which

involve numerous, intertwined problems. Breaking off and

working on small pieces of the problem can make the project

easier to handle. As more is learned about the user environment, more complex and expansive tasks can be undertaken.

• In early stages of a development, smaller investments of re- sources by users and developers can reduce the risks involved

in choosing one direction over another. The chances of a large

scale failure may then be reduced.

• Developments are likely to contain a number of changes which

can upset the flow of the project. Using smaller, discrete
543

tasks and gradually moving towards more complex undertakings

can help protect against major disruptions.

• Overall success may be more easily attained when smaller lines

of work are used. A strong track record of modest success may

increase the likelihood of overall success

On the other hand, smaller, more numerous tasks imply more project

control and monitoring than might otherwise occur. This could become

most inefficient if carried to an extreme. Developers have to balance

the risks in commiting themselves to certain task sizes with the

benefits they can achieve in work flow, control, and likelihood of

eventual success.

Factor 14 — Task Priorities

Description . The relative emphasis across tasks at a given point

in time and the ordering of tasks over time.

Discussion . The priority of tasks is another characteristic

developers should monitor throughout a project. Priorities include

both the emphasis across tasks at a point in time and the ordering

of tasks over time.

There are a number of factors which can affect priorities and

these should be considered when priorities are monitored. These

include

• Resource availability (When will money and manpower be available

for a task?)

• Time needed to accomplish a task (Is a task uncertain or well

understood?)

• Priority of user needs (What should be accomplished first

from the perspective of the user?)

• Interrelationships of subsystems (In what order must subsystems

be developed?).

• Need for accomplishments (What tasks will help build momentum

for the project and build credibility with the user?).

III. 3. 3 Team Personnel Factors

A second process characteristic developers need to monitor is
544

the operation of the system development team.

Developers must first ensure that the skills needed in the project

team are obtained. These will range from managerial to technical

areas.

Second, developers must monitor the turnover in personnel that

is likely to occur. Turnover of key personnel can slow progress

in a development by consuming time to transfer knowledge between

incoming and outgoing personnel, or by bringing in new perspectives

which must be accommodated with already existing strategies.

A final area developers should monitor is the commitment of

personnel to the project. Commitment of the staff to a project

is a predecessor to the more comprehensive commitments by users. Changes in these commitments, or differences between team members,

may be an early indication of uncertainties in project goals or stategy.

Factor 15 — Skills

Description . The availability of technical and interpersonnel

skills in project teams as required by system

development tasks.

Discussion . System developments require a range of skills which

developers must ensure are available. At one level, there are the

technical aspects of design which require people trained in specialized

areas. At a second level, there are interpersonnel aspects of design

which require people trained in understanding and working with people.

A development team should have a mix of these skills. It is also

important to have people with a mix of both technical and interpersonnel skills who can effectively handle the complex, intertwined

problems that developments involve.

Contracting for these skills may be necessary, and this may

present an additional number of problems developers should consider.

For example, contracting requires developers to design a procurement

procedure that specifies the kinds of skills needed. In addition,

contracting for support results in a new, outside team that requires
integration into the user organization.

Factor 16 — Turnover

Description . The change of personnel involved with project

teams.

Discussion . A major problem for a development team is turnover

in its personnel. Important knowledge can be lost in key project

areas that ensures continuity. Even in cases where no knowledge

is lost, newcomers will need time to familiarize themselves with

a project. This can cause delays.

Developers might prevent some turnover by initially obtaining

long term commitments from key team members who would represent a significant loss to the project. If contractors are involved with

the project, this is often done through the use of key personnel

clauses in the contract.

Changes are inevitable, however, and developers need also to

examine ways to minimize their impacts during a project. Several

actions may be taken:

• Developers can establish an executive management team, and

other teams for that matter, such that key tasks and

responsibilities are shared among several people.

• Another method is for developers to continuously encourage

people to remain throughout a development.

• Developers may want to minimze broad assignments for project

personnel.

Factor 17 — Commitment

Description . The team member support for and implementation

of the goals, strategies, and tactics of

the system development.

Discussion . Commitment of project team members to the goals,

strategies, and tactics of the development is an important predecessor

to the eventual full user commitment to the system. A lack of personal

commitment may result in key staff turnovers (see preceding section).
546

or be evidence of a much broader problem in the home organization.

Gaps in commitment between team members may be a source of confusion

and uncertainty which slows progress. Gaps between team members

and their home organizations may have the same effect. Developers

(team members) need to be sensitive to the commitments of personnel

in the project. Shifts or gaps need to be detected early and their

broader implications handled in line with project goals.

Several indicators of project team commitment are the following:

• The involvement of key, experienced personnel on project

teams from the various groups participating (e.g., from

the user organization, contractors, etc.).

• Top level management support from the home organizations.

• Time spent on the project versus time spent on competing projects in the home organization.

• Differences in commitment between personnel of separate home

organizations

111.3.^ Project Control Factors

Several factors having a significant effect on a system development are the processes used by developers to control the project.

Large scale system developments are complex undertakings which involve numerous activities, problems, technical issues, opportunities,

etc. They can be further complicated in situations where more than

one organization is involved in the project. Developers need to monitor

the control processes used so that revisions can be made if they prove

ineffective for accomplishing project objectives.

Three categories of process control methods are discussed in

this section. The first is the organization of teams and responsibilities. Developers need to establish a management structure which

matches the complexity of the development and the appropriate

level of distributed responsibility. The second category is the use of decision points and milestones. Developers can use specific points

in time or the accomplishment of selected tasks as control points

where progress can be reviewed, problems can be identified, or approv-
als for further work can be obtained. The final category is the use

of reports and reviews to monitor work, document and discuss

problems, or to disseminate information about the project to relevant outside stakeholders who may have influence over the development.

Factor 18 — Organization and Responsibilities

Description . The structure of project teams and responsibilities

in the system development.

Discussion . A key part of project control is the team structure

used by developers to manage and conduct the project. Part of the

structure should include a central authority which can coordinate

and manage the numerous organizations and people involved. Another

part of the structure will involve the distribution of responsibilities

to the personnel actually performing the work. Day-to-day control

of project activities will be located at lower management levels.

Developers will need to examine early in a project the kind of structure they use to conduct and control the work and then progressively

monitor its effectiveness in accomplishing project tasks.

Many sources advocate the use of a small central team for

managing a systems project. This team controls development overall

and reports to higher level decisionmakers in the organizations supporting the project. In projects where only one organization is involved,

such as with an in-house development, the team should probably consist

of both users and designers with users retaining ultimate control.

However, the situation may be more complex when mulitple organizations are involved, such as when contractors are used. While central

authority should still reside with the user organization, much of

the project work may occur outside of the user organization. Thus,

to some extent, control must be more distributed. A structure must

be established that provides a locus of control while ensuring adequate

responsibility to outside groups.

Another issue is whether developers utilize new teams or existing

ones to form the working level structure. Developers may choose
to form new teams to conduct the work by drawing on personnel from

existing organizational units. An alternative is to rely on already

existing groups or structures. In this case, project tasks would

be assigned to the units who would then decide how to perform the

task and staff up for the work.

Whichever approach is taken, developers will have to examine

how responsibilities are delegated to the working level teams. Central

ized control of all project activities will likely be impossible, and

for that matter, inefficient and unnecessary. Numerous problems

and decisions can be delegated to lower level working groups where

they can be effective handled as they arise. It will be advantageous

overall if working level groups can gain the sability to recognize

problems and opportunities on their own and then have the flexibiity

to handle them as appropriate for project objectives.

It should be recognized, however, that some problems (or opportuni

ties) may arise which fall outside the organizational lines chosen by

developers. Developers should consider the need for ombudsmen who can

be assigned to these unique problems. These individuals might report

directly to the central management team.

Factor 19 — Decision Points and Milestones

Description . The structure of specific events created by project

managers to recognize or review progress and

decide future courses of action.

Discussion . Large scale developments involve numerous activities,

problems and opportunities which need some level of ongoing recognition

and approval from the central management team. Developers first

need to examine in advance where major and minor decision points

should be placed and then actively monitor their occurrence.

Major decision points should be located at the end of each phase

to mark the conclusion of the activity and to select the strategy

for the next phase. They can also protect against making commitments

too early (or the fly before buy strategy).
549

The exact placement of major decision points is fairly arbitrary,

however. Activities implied by the name given to a particular phase

probably recur throughout the development of a system, particularly

if the progress on different subsystems varies or if cycles are made to refine earlier designs in light of new information about

the user. Developers may have to select points at which the development is, in general, making a transition.

Major decision points at the end of phases are not the only

control points needed in a project. Developers will also need to

emphasize sequential approval and acceptance of progress during each

phase. Problems and opportunities can be identified more quickly

and acted on at a point when action is needed. Reworking, redesign,

and argument after the fact can be avoided.

Within each phase, developers should thus select events or milestones as formal, recognized points for monitoring. Milestones can be

either points in time or events. They might represent a series of

events which conclude with some identifiable accomplishment. The key

is for developers to select milestones which provide insight to project activities and prediction power for future sucess or failure.

Milestones should be:

• Objective, in that they are not subject to widely different

interpretations

• Material, meaning they can be seen directly, and

• Significant, in that their accomplishment has some significance

in the project.

A series of milestones can have the additional benefit of leaving

a better documentation trail and improving the decisionmaking conducted

periodically at the end of phases.

Factor 20 — Reports and Reviews

Description . The structure of written and oral communication

mechanisms used to document and review

development progress.
550

Discussion . Decisionmaking and control of a complex system

development will involve some level of information transfer between

the various groups involved. Transfer and liason with other groups

outside of the project may also be needed. Two common mechanisms

for processing and transferring information are reports on project

activities and periodic reviews of major technical and organizational

issues. Developers need to structure in advance where these reports

and reviews occur and who will be involved. Developers then need

to monitor whether these mechanisms prove to be sufficient for transferring key information of the level and at the timing needed to control the project.

Developers need to consider several factors when designing or scheduling written reports. First, reports require a significant

amount of time to prepare. They can distract key members of the

development team from the priorities of the project. Developers

should keep the number and extent of reports small and monitor whether

too much effort is being diverted to their production.

Second, reports may serve several purposes beyond project control

which need to be considered when reports are designed. For example,

developers may want to document portions of the project for later

review. This might be important in areas where uncertainties remain

and developers choose to forego further work; later exploration could

be more fruitful. Reports might also act as marketing documents

for the project and the system.

Third, developers need to consider the appropriate level of

reporting. In some cases, more informal liason between groups may serve the purpose of transferring the needed information, while in

other cases more formal communication through channels is needed.

Finally, since reports take time to prepare and revise, their

utility in real time project control may be limited. Developers

may need to rely on other means to help control the project (see

design reviews below) and let reports act more as a record of already

recognized and approved activity.
More direct real time control may be exerted through the use

of face-to-face meetings and discussions which can quickly pinpoint

problems and options. It is common practice in system developments

to use design reviews for the purpose of reviewing general problems,

strategies, and the like, and making changes.

The basic strategy of these reviews is to compare progress,

either in specifying a design or actual construction, with previous

design plans. Inspections, for example, mean comparing hardware

with the paper design. Demonstrations can mean the review and accept

ance of assembled subsystem components in comparison to expected

overall system performance specifications. Reviews like these could

also be extended to more narrow, specific problems or subsystems

and occur more frequently than once during a phase. Developers need

to identify what reviews are useful and who should participate and

contribute to them.

III.3.5 Interaction with the User Environment Factors

A key part of system development is the interaction developers

have with the user environment. Interaction is needed in order to

transfer information about the user problems to the design teams

and to progressively develop and transfer the system to users. Inter

action is also beneficial for establishing relationships with users. Their openness and support will be critical for design and eventual

institutionalization.

There are two related perspectives developers should monitor

about interaction. First, there is direct user involvement in the

project. User involvement is usually one of the main factors found

to be important to project success. Second, there is contact with

the user organization in general. Extensive contact should be made

in order to extract the key user problems around which the system

needs to be designed, verify concepts and system performance, and

to transfer the finished design. Sections on each of these processes

are presented below.
552

Factor 21 — User Involvement

Description . The participation of members of the user organization

in the managerial and technical teams of the project.

Discussion . One of the major factors found to be important

in system development is the involvement of users. Projects conducted

in isolation from users have often resulted in designs which do not

meet user needs and gain little support for institutionalization.

Developers need to identify areas where user participation is required

or beneficial and monitor tasks to ensure that participation is solicited and utilized.

One approach to user involvement is to include users on project

teams, both at the managerial and working levels. This provides

a mechanism for user control of the project and a basis for eventual

ownership of the system. It also helps ensure access to the organization for the collection of information needed by design teams.

Developers should also consider involving users from different

levels of the organization. Managers and line groups may have different perspectives on the information problems and the appropriate

designs.

Obtaining user involvement may be a difficult problem in itself,

however. Users may not have the time to be actively involved or

it may be unclear who should be involved, particularly if the system

is a new entity for the organization and the final home is unknown.

If there is little concensus on the need for a system, users may

also resist involvement and continue to support other systems.

Close involvement may also have some risks which developers

should consider in soliciting participation. Early project stages

are likely to be characterized by exploration, uncertainty, and shifting concepts. Utilizing the limited time of users at this stage may

be inefficient until clearer tasks emerge. In addition, high levels

of involvement may raise expectations about the system. Difficulties

or delays may lead to higher disappointments later if these initial

expectations are not met.
Factor 22 Problem Identification

Description . The overall amount of attention to and the amount

of contact with the user organization in the

definition of user problems.

Discussion . One of the important parts of the development process

is problem identification. Information problems are likely to be

unstructured and complex. They will probably be closely intertwined

with other problems in the user organization. Early definitions of

the problem may be too general or simplistic and, in any case, will

probably change as more is learned about the user environment and

design options. Attention to problem definition should be given

early and routinely throughout a project.

A systematic approach to problem identification will benefit

the development. Emphasis should be placed on learning about the

user environment from a number of different perspectives: how it

works, goals, decisionmaking styles, decision processes, information

flows, other information systems, etc. The goals and expectations

for a new system would also be an important item to include here.

Developers should conduct analysis at the highest decisionmaking levels

and work into the organization, while concurrently examining the

lower working levels and moving up decisionmaking chains.

Problem identification should also be extended to the external

environment of the user. The user organization will probably have

relationships with outside groups that can have significant effects

on the organization itself. Developers should examine and monitor

these linkages and determine how they might affect system design

(see also Boundary Factors above). It would also be beneficial to

determine what information about the system these groups might need

during the development (see also Reports and Reviews, Factor 20).

As information is gathered from these areas, it may become useful

to organize it by constructing some models. At first these might

be descriptive models about what is happening: the players, the

decisions, and the information needs. Later, a second set of normative

models can be developed to describe the decision processes users would

like to have. Descriptive and normative models can then be compared
to help identify key information problem areas. Developers should also consider how the information they collect

during problem identification can be used to help the organization

directly. A system development may present one of the first times

the organization has been closely examined, particularly by outsiders.

Developers may uncover factors about the organization that were not

previously known. Detailed feedback to users about current processes

may identify improvements which can be made immediately without proceed

ing with a new design. Close study may also reveal widely varying know

ledge or perspectives on organizational problems, including the informa

tion problem of concern. It may be important to project success to

reduce these differences. User recognition of the need for change has

been found to be a key factor in successful developments. Developers

should spend time communicating and selling the problem to the user. Spending time on problem definition with users may also have

additional benefits besides locating the need for a system. It will

begin to set a precedence for close user contact and involvement

with the development. Some users may not understand what a systems

project is or will have significant anxieties over how it will change

their jobs. They may resist providing information or access to their

programs. Similarly, developers may be uncertain of what will happen

or be inexperienced in what to do. Close contact between these two

groups can help reduce these problems by promoting a mutual understanding.

Other benefits might include the following:

• Establishing a language about problems and designs which users

and developers understand.

• Identifying key underlying assumptions in problem definitions

which need examination.

• Discovering areas where resistance to change may be great

and organizational politics important.

• Starting to build user commitment to the system.
555

Factor 23 — Testing

Description . The amount of concept and design testing conducted

in the user environment.

Discussion . As more is learned about the user information problems and the user environment, concepts and designs for the system will

emerge. Eventually, it may become necessary to directly test designs

in the user environment as a means of substantiating design claims.

Testing in the user environment may also be important and beneficial

for:

• Competitive demonstration.

Several competing proposals for a system design may emerge

in the project and testing in the user environment may be

critical for selecting among them.

• User involvement.

Testing in the user environment can give users something

to react to, especially those users not directly involved in

project teams. Their knowledge and opinions may be critical

factors for system designs. Participation in a low risk test

may be an effective means of obtaining their ideas and support.

• Commitment.

Favorable results from tests may be one important factor which

eventually results in full user commitment to the system. In

addition, opportunities may arise for early use of the system

while it is still under development. These may provide an early payoff which should be taken advantage of.

Developers should establish an active program of testing with users

and continue it as long as necessary in order to resolve uncertainties

in design.

There are several levels of testing in the user environment

which might be considered by developers. One is to use models on paper or dummy mock-ups of the system and/or its products. The earliest of these might be descriptive and normative models of user decision

processes which evolve in problem identification activities (see above

section). Rapid feedback to users of information like this can further

educate users as to what is needed as well as provide a test to determine whether developers are understanding their needs.

At another level, working models of subsystems might be constructed and pilot tested under various conditions with users. Different
parts of the overall design can be tested by themselves and at the

appropriate stage of development. For example, developers might

choose to focus on a particular decision problem and then

produce a working model to test their design concepts for this one problem.

Finally, as many of the system components become developed,

a prototype system can be formulated and tested with users. A prototype design would be expected to include most or all of the functions

developers had found to be needed by users. It would also include

the various interconnections between subsystems or between the system

and outside groups. The prototype design would provide for a full

scale system test and check the emerging design against the original

performance objectives.

An important part of testing at any of these levels will be the

development of user criteria to evaluate the performance of designs.

Developers will need to identify in advance and progressively refine

the criteria users prefer to apply. Some of them may be appropriate

for institutionalization within the system for routine use after

development is completed (see also Evaluation and Updating, Factor

12 ).

Factor 24 — Transfer

Description . The amount of attention devoted to the transfer

of the system to the user environment.

Discussion . A key transition point in a systems project is

the transfer of control and support from the developer to the user. Transfer of the system to the user is often referred to by several

names: implementation, institutionalization, cut-over, or conversion.

Developers need to think through the implications of changing to

a new system and routinely monitor whether their strategy is leading

to a smooth transition of ownership and full user commitment.

Transfer of the system to the user is an activity which can and should begin early in a project. As concepts and pieces of the

design evolve during problem identification activities, developers
557

should consider how the designs would become fully operational and

institutionalized within the organization. Examining the differences

between the new design and existing practice will help identify

possible transition strategies and key problems (see also Novelty

of the Design, Factor 11). Close user involvement in this analysis

is also important, both to identify issues and to initiate the

transfer of concepts.

Later as designs are detailed, testing provides another stage

in which transfer considerations should be studied by developers.

Tests in the user environment may be designed to include user participation in using and working the system. This will provide a temporary means of exploring how to transfer system components to users

from the developer perspective. It will also provide a low risk

means for users to explore the transition to a new system. Feedback from their perspective will be useful. Developers should be able

to learn lessons about transfer from testing which can be used later

in future tests and final institutionalization.

When it is appropriate or required, the system or individual

components will be transferred permanently to users (the turnkey

system) . Developers may have the choice of transferring the system

gradually or quickly making the transition from the old to the new

system. Several factors can be considered in deciding which

approach to take:

• Abrupt transition.

Complete rapid transition to a new system may be too risky

or unnecessary. If there are user problems needing quick

attention, it may be beneficial to institutionalize key com- ponents as they are available. Radical changeover may create

a number of side impacts with users that only complicate the

transition and start-up periods.

• Backup systems.

It may be beneficial to gradually phase in the new system

while gradually phasing out the old. This will provide some backup or protection should unexpected problems occur. Users

should not be left without any system if delays occur.

• Training.

Users need to be trained to work with a new system. Users

should be involved in developing training programs and training needs should be a part of design activities. The availabil-
ity of these programs may influence the transfer point.

• Documentation.

Documentation of the design and related matters should

be complete, especially before transfers are made. Areas

where design problems remain should be thoroughly documented

so that users can continue development later.

• Management support.

As the point for transferring a system to users approaches,

it is possible that managers (both of the project and the user

organization) may start assuming that the system is complete

and its implementation routine. This may cause a lack of

attention towards transfer which jeopardizes final stages.

For example, lower level and less experienced staff may

be assigned to transfer tasks and cause implementation

failure

When the system has been mostly or completely transferred to

the user, a final stage of withdrawal and termination should begin

for the project teams. In termination, developers need to ensure

that

• Ownership and control of the system rests with those who must

use and maintain it.

• Necessary new patterns of behavior have become a stable

part of the user's routine.

The basic goal is to refreeze the organization by removing the disturb

ances of change and leaving the organization in a stable position. De

velopers should ensure that evaluation and updating functions become

operational so that users learn to revise the design themselves (see

also Evaluation and Updating, Factor 12).
559

III. 4 User Commitment Characteristics (Factors 25 - 30)

111. 4.1 Introduction

The final category of factors in the monitoring framework is

for measuring the extent of user commitment to the system. Commitment

to the system is viewed as a series of user decisions or actions over

time which indicate increasing user interest, support, and acceptance

of the system. Full commitment to the system will emerge when users

elect to assume total responsibility for the operational system and

the development project is terminated.

This section identifies and describes six factors for developers

to use in monitoring commitment. These have been divided into two

areas: factors relating to use of the system and factors relating

to support of the system.

111.4.2 System Use Factors

Major indicators of user commitment to a system are the actions

taken toward use of the system. In early project stages, these user

actions may be decisions about the design and application of the

system to organizational processes. Later, as the system is developed,

these actions will change to use of the prototype system or subsystems.

Developers need to closely monitor these actions so that changes

can be made in the design or the development process if needed.

Three factors are described in this section. First, developers

need to identify the types of applications of the system (or ones

it is expected to have). This requires identifying system actions

with specific user tasks. The second indicator of use is the activity

resulting (or expected to result) from system applications. These

actions need to be documented as they will likely play a key role

in obtaining user commitment to the system. Finally, developers

should monitor the extent of use in order to gauge how frequent and

widespead it is (or will be). This may be an especially important factor to the top level managers who will decide whether to provide the

resources needed to fully support and institutionalize the system.
Factor 25 — Applications

Description . The use of specific elements of the system in

specific units of work of the user organization.

Discussion . Developers need to identify the applications of

a system to user problems as one part of monitoring user commitment

to the system. In early project stages when designs are being formulated, developers will have to rely on the expectations and plans

about applications rather than actual cases (see also Boundary

Factors above). These plans can provide important insight, however,

to the extent and type of commitment users expect to make. Later, as system components become operational in testing and implementation

stages, developers can then monitor whether the applications are

in fact occurring and are leading to full institutionalization

of the system. Problems or changes in applications may indicate

a need to revise the design or project strategy.

It is important that developers take a broad view of what constitutes an application. This primarily means that definitions of use should not be restricted to simply identifying whether a major

decision is the result of system use. Numerous applications of the

system may have occurred to support a decision process even though

a major decision is not evident.

Developers should also include unexpected applications and areas

of non-use. Users may perceive that the system will not (or actually

doesn’t) solve their problems. They may turn to other systems and

withdraw their support from the project. Developers may also find

that staff who operate the system are having difficulty doing so. They might prevent or seriously impair further operation in an area unless the design is changed or they receive more training. User

resistance at these levels may have serious implications for eventual

user commitment to the system.

The key problem in determining where applications occur (or

will occur) is to match some unit of work in the organization, such

as a decision, with some unit of system action. This matching can

be greatly facilitated by the use of models which detail the informa-
61

tion and decision flows of the user organization. Developers can take

these models and identify places where the system interacts with

the user. The range of applications can then be determined as can

the specific user behaviors which define work tasks.

It is advantageous to start modeling early, such as during initial

problem definition stages when extensive contact with users occurs

(see also Problem Identification, Factor 22). Developers can then:

• Obtain an early indication of commitment by matching expected

system products and applications. This can be compared to

project goals (see also Boundary Factors).

• Plan the occurrence of applications as the project proceeds,

possibly starting with simpler ones and then moving into more

complex undertakings (see also Task Structure Factors).

• Obtain a basis for establishing measures of actual use which

can be applied later during testing and implementation

(see also Project Control Factors).

• Uncover any different perspectives as to what constitutes

use. Differences among users may prevent agreement that an application has occurred (or will occur) and have an effect on eventual institutionalization (see also Boundary Factors).

Factor 26 — Consequent Actions

Description . The actions resulting from areas of direct system

use. Discussion . Further indicators of use are the user actions

which come about as a consequence of system applications. These

actions should be observable user behaviors which can be documented

and shown to others. Changes in thinking may be a valuable result

of system use, but these are hard to measure and probably would

be considered weak evidence to justify support for the system. Credible evidence is of critical importance to both users and developers.

User managers need proof that a system improves the operation

of their organization before they provide full support. Developers

need to evaluate performance and commitment and then determine whether

changes are needed in design or strategy.
562

There are several categories of actions developers might consider

monitoring in order to measure the effects of system use. These

include

• Follow-on actions in organizational processes.

Developers can identify the linkages of various organizational

actions and trace the impact of system use down the line.

• Organizational changes.

Use of the system may eventually bring about changes in the

organization, such as in structure or official procedures.

Changes may occur in other systenis or processes which support

system use (see Changes in the User Organization below).

• Performance changes.

Factors which are used by the organization to measure performance might record changes which coincide with the introduction

of the system. Performance improvements may consequently lead

to high level commitment to the system.

• Avoidance or non-use of the system.

Negative experiences in a particular application may lead

some users to avoid further contact with the system, discontinue their support, or work against the system (see also

System Champions below)

As with identifying system applications, determining where actions

like these occur and then measuring them can benefit from initial

detailed modeling of user processes. Models can help identify the

flow of information or decisions and provide developers with a means

of tracing user actions which emerge from specific system applications.

Identifying these actions in advance also provides a means of determining what decisionmakers consider valid evidence of use. In addition,

expected actions identified early through modeling can be compared

with actual behavior during use and provide a means of measuring

changes in user commitment.

Factor 27 — Extent of Use

Description . The amount of use relative to the number of potential

applications

Discussion

The third indicator for monitoring use is the

extent of system applications to user processes. Measuring extent
includes determining the frequency of use and the amount of use relative to the number of potential applications. These factors

should be monitored over the life of the project and used to indicate

changes in user commitment. Stable or decreasing levels of use might

indicate dissatisfaction with the system design or loss of momentum

in project strategy. Revisions in both areas may be needed.

One major question that developers and users alike have to

consider is how much use constitutes acceptance of the system. User

top management will likely have some threshold for the extent of

use above which they will give serious consideration to full support

and institutionalization. This threshold may be based on how often

the system is used over the range of different applications as well

as how this use affects the performance of the organization. Developer

can begin to uncover thresholds like this early by spending time

detailing the types and extent of applications decisionmakers expect

to achieve. This may be best done by, again, modeling user processes

where the system is to operate and determining what use specifically

means. Developers should then continue to examine the extent of use as designs are implemented and determine whether the expected

levels are achieved and prove to be significant enough to justify

full institutionalization.

III. 4.3 System Support Factors

Besides monitoring the direct use of the system to determine

user commitment, developers should also monitor other user actions

which indicate support for the system. These actions may include

early signs of user acceptance such as the emergence of user advocates

as well as longer range institutional changes which are needed to

support the operation of a system. The lack of supporting actions

like these may indicate that the system is isolated from the user organization and that it will not become fully implemented when the

development ends.

Three factors for monitoring system support are described in

this section. These include the emergence of system champions, the
resources allocated by the user during development, and the changes

in the user organization which are needed to support system use.

Factor 28 — System Champions

Description . The emergence of advocates for the system in

the user organization.

Description . An important source of support for a system development can be a critically placed key man, advocate, or system

champion. A system champion is a person, perhaps best located in

the user organization, who believes very strongly in the system:

its concepts, design, performance, etc. The system most likely

matches the champion's objectives and the organization's objectives.

A champion has credibility inside the organization, hopefully across different levels. His credibility may even extend outside the

organization as well. He can push the system into existence and

also find the pull for it from users. He should also be able to

isolate the system from attack by others.

Developers should seek out system champions and cultivate their

participation. Their help can be a critical element for success

throughout a project, especially in early stages when both the project

and the system are being defined. System champions may offer the

earliest sign of user commitment and act as a catalyst for acquiring

user support.

Champions may emerge from different levels of the user organization

and developers should actively seek this broad-based support. Champions

from top levels of management are clearly important to a development

since they may have great influence over resources, access to the

organization, or project strategy. However, developers should seek

champions at other levels as well. This can help gain credibility

for the system with key groups and also expose project teams to different perspectives or expectations about system design or project strategy. In addition, multiple champions, either within or across organizational levels, may reduce the potential disruptions caused by the loss

of any one of them.
Developers must be skeptical, however, of champions who may

be overly supportive or controlling. They may be attempting to capture

the project, and thus the system, in order to use it for their own personal rather than organizational objectives.

Developers must also be aware of those who actively oppose

the system. Just as the new system may have a key advocate, it may

also have someone who is uninterested or critical. Opponents may

fight development at each stage or build up opposition. They may

also elect to hamper an operational system as much as possible or sabotage system processes. Developers should be open to the possibility that opponents exist, or will develop, and be prepared to actively

counter the effects of their actions.

System opponents may not be limited to those who oppose development outright from the inital planning stages. Technical problems, infighting already existing in the organization, differences of opinion

over designs, or opposition to particular project decisions may

create new opponents during a development. In these situations,

it is critical that developers direct specific attention to the resolution of conflicts and consensus building. System developments can involve change and disruption to users, and the creation of an active

opposition only makes strategy that much more complex and success

that much more difficult to achieve.

Factor 29 — Resource Commitments

Description . The type and extent or resources allocated

by the user organization to support the system.

Description . A key indicator of user support for a system is

the type and extent of user resoucres committed during the development.

The two primary resources users can provide are manpower and money.

In early development stages, these might be supplied to the project

to support design and testing. Later, as subsystems become available

for implementation, user resources may be supplied to operate them.

Each of these allocations is a sign of user commitment to implement

a system after the development is completed.
566

There are several characteristics concerning allocation of

money to a project which developers may want to monitor. These

are

• Temporary versus permanent allocations.

In some cases, part or all of the development may be funded

out of special user project funds and be combined with

resources originating from outside third parties. A change in

budgeting, where the system becomes a line item in a user budget,

may be an important indicator of emerging support.

• Origin of money.

Developers may want to compare the origins of financial resources

with the location of primary users. Arrangements not following

organizational lines could mean broad support or potential ownership problems. In addition, if multiple sources are involved, developers may want to consider the relative proportions of allocations versus expected use. Continuing support

from all parties may be essential for full scale system implementation.

• Planned allocations/expenditures.

Developers may want to examine how user allocations are planned

over time and in what system areas tiiey apply. Targeted

resources may signal important areas where user commitment

will be based. User expenditures should also be monitored.

Success in the development will most likely be followed by

increasing resource allocations.

These factors may not apply in projects which are entirely based

on third party funding and where user resources are expected only

after successful development.

Similar characteristics can also be monitored for user allocations

of manpower. They may apply independently of any funding allocations

and be essential for strategies involving close user involvment.

Developers should consider:

• Temporary versus permanent assignments.

User staff may only be temporarily assigned to project teams

or to the operation of initially available subsystems. While

these conditions would indicate an initial commitment, a transition to permanent assignments to operate the system

components would indicate a more significant level of support.

• Type of personnel assigned.

The development may require the skills and involvment of users,

and developers should monitor whether they are made available.

The assignment of key people in an organization may signal
567

a high degree of user commitment to the system (see also Team

Personnel Commitment, Factor 17).

• Level of involvment.

A key sign is also the extent of involvement. User personnel

assigned to work on the system a small part of the time may

be too distracted by other business to contribute effectively

to project objectives. Allocation of significant portions of

staff time may indicate a solid conmitment to the system.

Commitment would also be indicated by increases in the proportion of their time.

Factor 30 — Changes in the User Organization

Description . The alteration of policies and procedures in

the user organization in order to support

system operation.

Discussion . Other indicators of user support and commitment

to a system are the changes which take place around the system.

Systems are placed into complex organizations involving intertwined

line of communication, support, responsibility, and activities.

Changing over to a new system likely causes and/or requires changes

in other places and sytems. New relationships and functions may

be established. Developers should look beyond the immediate areas

of system impact to other areas where change may indicate a positive

or negative force for user commitment.

There are numerous potential changes inside the organization

which developers can monitor. Some may occur in the areas which are

to support the operation of the system. For example, the system

may require new types of personnel or new positions which must be

arranged for by personnel divisions. Some users, may need training

in order to operate the system and new programs may be created for

this. Another similar change is the creation of an incentive system

either to attract users to the new system or to acknowledge improved

performance because of its use. Other changes may occur in processes,

such as decisionmaking style or how an organization works. Changes

may also be observed at higher management levels. For example, the

system may support a new policy which high level decisionmakers must

acknowledge and support. Finally, developers should examine changes
in competing systems. The continuation of competing systems may

indicate a lack of support for a new system, while conversely, the

gradual withdrawal of support from them may indicate the opposite.

Developers should also monitor actions or conditions outside

of the user organization which may affect user commitment to the

system. For example, outside institutions may mandate the use of

the system. Outside organizations may also have a role as system

users or suppliers of information. Changes in their roles or activities may be a direct or indirect sign of system acceptance and support

in the user organization.
IV. The Monitoring Functions

The following sections describe the five functions monitoring

can serve in a system development. As described earlier, the monitor

ing process has been designed to serve both administrative and re- search purposes.

The five functions have been designed to overlap and support

each other for these uses. These functions are:

• Problem identification — The tracking and assessment of key

areas where development problems typically occur, both in the

short and long range.

• Strategy development — The identification and development

of explicit management actions to solve problems and revise

strategy.

• Research — The design and implementation of real-time studies

of the development process.

• Documentation — The establishment of an organized and stable

recording process which can identify special or recurring prob

lems, and support decisionmaking and research.

• Dissemination — The distribution of key development information to these inside and outside a development as a means of

facilitating coordinated actions and distributing knowledge

gained.
570

IV. 1 Function #1 — Problem Identification

Problem identification is the major function for the framework,

reflecting the ETIP management need for a procedure to anticipate

and pinpoint key problem areas. The framework is structured to contain key elements of a systems project, areas where problems usually

occur and which are important to eventual project success. By routinely monitoring these areas, managers can identify problems more

quickly and possibly earlier than would be the case without a framework. Use of the factors may also promote easier recognition of problems. In addition, problems which are new, complex, or multi-faceted,

may be more easily decomposed into recognizable, manageable components

by analyzing them with the framework factors.

IV. 2 Function #2 — Strategy Development

The second major function for the framework is strategy development. The performance of a strategy can be monitored through the

elements of the framework. Problems and progress identified in these

areas may highlight the need to change some or all of a strategy in

order to continue progress towards objectives. If strategy problems

occur, the factor structure and the information base available from

regular monitoring may then also assist in formulating the changes

needed. In addition, the framework can promote better strategy by

offering a means to consider longer term pespectives. First, retrospective analysis of the factors may uncover problems that are unidenti-
571

fied by constant attention to immediate circumstances. Similarly,

by focusing on specific framework factors during strategy revision,

managers may be encouraged to predict the effects of strategy changes

in the future. Besides bringing a longer term perspective to problem

solving, this can provide an opportunity to identify the conditions

or points in time when a strategy should again be reviewed. The longer term cause and effect linkages which are identified through these

analysis may also help managers avoid repeating mistakes.

IV. 3 Function #3 — Research

The third function of the framework is to provide a means for

researchers to study and contribute to on-going system developments.

This reflects the finding that systems researchers see a need for

closer contact with actual projects — partly to gather empirical

evidence not easily available through other means and partly to transfer the guidance available in the literature to practitioners who

some claim have difficulty using the literature. While these two

problems are to some extent exacerbated by the few incentives practitioners have to study their own experiences, it does appear that an

approach which unites researchers and practitioners may be a good

way to gain the access needed by the research community.

The monitoring framework provides a research opportunity by establishing a data collection process that researchers can use to define

and implement studies. By adapting their studies to the framework

structure and procedures, researchers can acquire a series of data
points at the same time as project managers. They can then use this

data to study changes and linkages between factors of interest and

to produce valuable insights of use to others.

Researchers may find it beneficial to design their studies with

the project managers. Besides helping to ensure continuing access

to the project, this approach may provide researchers an opportunity

to help managers with problem solving. Research studies could be

designed to support problem solving. In addition, researchers might

be able to bring in relevant guidance from the literature to help

managers. This would help counter the claim that research in the

systems literature cannot be used.

IV, 4 Function #4 — Documentation

The fourth function of the framework is to document project activi

ties for the administrative and research purposes discussed above.

Documentation here means establishing an organized and stable written

record of project activities as they occur in the key selected areas

of the framework. In the short term, this function can help project

managers identify current problems and progress and assist in finding

ways to improve the project. For researchers, documentation of project events as they occur is one essential feature of the monitoring

approach that makes it attractive over retrospective studies.
573

I

I

i

Documentation is seen as particularly useful, however, for tracing and analyzing selected factors over long periods of time. For

managers, documentation of project activities can be inefficient for

short term problem solving. In addition, documentation over the long

term can provide several benefits that may be difficult to obtain

from short term analyses of immediate actions:

• Routine documentation over the long term can help managers

identify special, evolving, or recurring problems.

• Longer term analyses of management actions may help identify

successful and unsuccessful approaches, promoting improved

strategies or new ideas for current actions.

• Long term documentation can support decisionmaking in the

project, particularly for major decisions at the end of

phases. These decisions may rely on credible evidence of

performance — unattainable from quick retrospective analyses

at the time of the decision.

• Documentation can be used to inform new staff or project

history and thus ease the problem of turnover.

For researchers, the long term stream of data available from

documentation provides the data base on which to conduct research.

In particular, long range documentation provides;

• The evidence upon which to identify and test relationships

between important factors in development.

• A basis of developing models of the dynamic, evolving process of a large scale system development presently missing

in the literature.

• Empirical evidence about all project stages that is also

needed in the literature according to some researchers.

This may be especially important for other researchers who

need better access to actual projects.
574

IV. 5 r^unction #5 — Dissemination

The fifth and final function of the monitoring framework is to

promote dissemination of information about the system development

to groups not directly involved in oversight roles. The framework

structure and resulting data base can facilitate the transfer of information to these groups by making it easier and quicker for managers

to generate the information needed. This of course must be tempered

with the need to avoid release of interim or uncertain information

which might harm the project.

Managers may find the dissemination function useful both inside

and outside a project. For insiders, dissemination of monitoring

information can facilitate a common awareness of problems and progress

and help promote united actions. In a large project, this may be

especially important in coordinating groups working on different parts

of a system. Dissemination of pertinent information about progress

may also have the secondary effect of promoting or facilitating the

monitoring activity. Project staff may come to rely on periodic reports of activities for guiding their own work.

For outsiders, dissemination of monitoring information can be

essential to keeping groups informed about progress and building awareness and support for the system. This may be especially critical

to user groups not directly involved in the development, but having

some indirect role in future system operations. Other important groups
can be top level managers or outside institutions who supply resources

to the project, periodically review progress, and approve continuations of the work. These people may need current and retrospective

reports on development activities, both easily supplied if monitoring

has been routinized.

Dissemination is also important to researchers. By providing

the opportunity to closely study on-going development activities,

the framework can facilitate the transfer of information to the systems literature. For example, the framework can help provide a more

dynamic, evolving view of development and other empirical evidence

that some researchers believe are needed in this area. Besides helping other researchers, real time analysis of project activities may

offer insights to other system developers who are looking for guidance

of use in their own situations.
576

V. A Model for Implementing the Monitoring Process

V.l Introduction

The framework has been designed with a set of procedures that

managers can use to apply it to specific developments. This process

is modeled in Figure IV. 1 by a flow diagram of events which are gener

ally expected to be a part of monitoring. The model is illustrative;

modifications would likely be needed in order to match the process

to the project.

V.2 Implementation Procedures

As shown in Figure IV. 1, the monitoring process begins when project managers identify the need for monitoring and proceed to specify

the indicators, procedures, and staff needed to operate the monitoring

system. To specify the monitoring indicators, managers need to examine the individual elements of the framework and decide what indicators are needed. For example, under the category of user commitment,

managers need to identify events that will indicate use and support

for the system. For the system champion factor listed under support,

this will involve identifying where champions would arise and what

actions will be considered supportive. This identification process

will of course be influenced by what managers and others, inside or

outside the project, need to know during a development. In addition,

managers may find gaps in the elements or alternative factors which
577

FIGURE IV.

A GENERAL MODEL FOR APPLYING THE MONITORING FRAMEWORK
578

2 better match their specific situation. Analysis of each framework

element in this manner will specify where monitoring should occur

and what needs to be monitored.

Next, managers need to specify the procedures for monitoring.

Some of the considerations include the following:

• How monitoring will be coordinated with major decision points

in the project.

• How monitoring data will be collected.

• Who will receive monitoring information and what information

will be supplied.

• The storage and location of monitoring information.

• What kind of documentation is desired.

• How research activities will be tied to management activities.

Operational details in these areas will establish how monitoring is

to be performed and integrated into project management.

Finally, at the same time as these procedures are being explored,

managers should also identify the team which will perform the monitoring. While top level project managers are the main users of monitoring information, their role in collecting the information should be

minimal. This work should be delegated to the staff supporting project

leaders. Managers will have to decide who will be on the team and

2 The framework is intended to be comprehensive. However, this is

still the first attempt at defining such a framework. Revisions and

additions are expected.
579

how the different groups in a development should be represented.

It will be beneficial if the team consists of members from all participating groups: the user organization, the development team, and

the group of researchers (if present) studying the development. This

should help ensure collection and use of monitoring information, while

improving the credibility of the monitoring by bringing different

perspectives to the process.

After these initial preparations are completed, the monitoring

process is started. Problems and progress in the specific areas of

3 framework are identified as they occur or at selected periodic reviews.

If the situation is new or not well understood, the monitoring team

may need to decompose project events into smaller elements by identifying the relevant factors in each framework category. The relationships between different factors should also be considered. As appropriate, the monitoring team then produces a problem statement for

project managers. This procedure is illustrated in Figure IV. 2.

With the problem identified, managers then decide what, if anything, should be done. This is when the strategy development function

of the framework can be useful. If managers know what should be changed,

then no further analysis is needed and the changes should be made.

In this case, the only remaining tasks for the monitoring team are

3 It is assumed here that situations occur which require intervention

by project managers.
580

Identify Initial

Event of Interest

Problem Definition Using the Framework

Problem Components by Category and Relationships

Design Process Commitment

Relevant Factor 1

Relevant Factor 2

unexpected

relationship

Relevant Factor 3

\

relationships

Unexpected Factor 5 4

I

I

Irrelevant Factor 6

Relevant Factor 4

no relationship

Produce Problem Definition

For Managers

Based on Factors

FIGURE IV.

PROBLEM SOLVING USING THE FRAMEWORK
581

to: document the problems and actions taken for future reference,

identify future conditions when these actions should again be reviewed,

and disseminate information as needed about the management activity.

However, if the appropriate action is uncertain, further analysis

of the problem using the framework may be beneficial for identifying

options. In this case, the framework factors may provide a focus

both for the problem and the management intervention. Managers can:

• Identify the key components of the problem using the elements of the framework and what actions might be targeted

at these areas.

• Identify and examine the relationship between problem elements, using the different framework categories to decompose

a complex situation.

• Examine guidance on the problem elements from previous monitoring information, project research underway, or other

outside sources.

Once the interventions are specifically identified, the monitoring

team then determines how to trace the effects in the future and whether

special attention is needed to identify when a review of the changes

may be needed (see illustration in Figure IV. 3). Over the long run,

tying the state of the project to management interventions should

help the monitoring team identify successful approaches and promote

better management control.

After the changes are made and plans for future monitoring are

established, the monitoring team documents the activities and dissemi-
Uncertain

Intervention

582

I

i
583

nates information about them as needed to others inside and outside

the project.

This pattern of monitoring continues during the project until

monitoring is no longer needed by managers. At this time, the monitoring team completes documentation of the project, emphasizing the final

state of the project in terms of the framework elements. Researchers,

if present, then conduct analyses of the data collected by the monitoring team according to their earlier plans. The monitoring team should

also review the framework to determine how well it worked and whether

changes are needed for future applications.

Finally, the results of the project are disseminated as appropriate to other interested researchers and practitioners.
V. Conclusion

This manual has presented a process managers and researchers

can use to monitor important areas of complex system developments.

The manual has described a framework of thirty monitoring factors,

five functions monitoring can serve, and a general model for implementing the process.
585

APPENDIX C

Letters From Reviewers
586

(retyped for the dissertation)

April 8, 1981

Dr. Charles W. N. Thompson

Northwestern University

Evanston, Illinois 60201

Dear Dr. Thompson:

Steve Garrity has asked me to review his dissertation and provide some comments on his results. Since these comments might be useful in his

defense of the research, I thought I might briefly describe my reactions

to you in time for the meeting on April 13*

I have reviewed most of the dissertation — reading some parts, scanning

others, and passing over others (per his instructions). I have reviewed

most closely Chapters 1 and 7, the introduction and conclusions, respectively. I have scanned Chapters 2, 5, and 6, in order to review the

proposed monitoring framework and the specific ETIP cases which Steve

has used in its development and testing. I did not review either Chapter 3 or 4.

I have some overall observations about Steve’s work at this time. I think his work, and especially the proposed framework, are valuable

contributions to ETIP. I am very interested at this point in having

the approach implemented within the Regulatory Program over the next

several months. I have already requested that Steve begin planning

a series of briefings to ETIP, contractor, and agency staff as a means

of initiating this monitoring function.

I hope you find these comments useful in your deliberations on Steve’s

accomplishments in his dissertation.

Sincerely,

Daniel W. Fulmer

Group Leader

Experimental Technology Incentives Program
587

UNITED STATES DEPARTMENT OF COMMERCE

T!ia Assistant Secrstary for Productivity,

Technology and Innovation

Washington, O.C. 20230

(2021377-3111

- /

April 10, 1981

Dr. Charles W. M. Thompson

IS/MS Department

Technological Institute

Northwestern University

Evanston, Illinois 60201

Dear Dr. Thompson:

Steve Garrity had asked me to review key portions of his dissertation dated

April 13 and provide comments to him as partial preparation for his defense

next week. I have responded to Steve orally but he thought my comments relative to the utility of its results might be useful to you as well.

•Most of my career, industry and government, has involved system development

projects, primarily of major hardware systems. For example I was a key player

on the FAA team that directed the SST development during the 1964 through

1971 period. In industry at Martin-Baltimore I was an associate program director

of the Mobile Mid Range Ballastic Missile (MMRBM) program and played supporting roles in other major space developments as well. In my last assignment

before assuming my present assignment I was Associate Administrator for R&D

at the FRA and inherited many projects that were in trouble due to improper

system integration and project monitoring. As you can ascertain from the above, I'm familiar with large scale, complex system developments that are primarily hardware oriented.

Since taking command there at Commerce of the ETIP program, I have noted striking similarities between my past hardware related experiences and ETI? issues

and problems. In short I find the same system engineering/management approach

applicable to both.. More to the point, I find at ETIP ample evidence of a lack of front end system planning and a management process that assists ETI? staffers to effectively monitor and stay ahead of program problems.

In this respect Steve's framework contains the key elements I have found to work successfully in staying on top of complex systems.

1. Task structure breakdown - or WBS-permits tracking of resource commitments and schedules for areas that have measurable outputs (tasks must be selected to meet these criteria)

Early user input to define outputs and determine champions.

General periodic monitoring to relate output performance, resource

application, and schedules at task levels.
4. Detailed monitoring of those areas where either costs, schedule

or outputs start to show significant deviations - this is the firs

sign of problems and is key to a living planning process.

5. Broad, experienced project monitoring people to do above.

I haven't gone over his entire list but they either fit directly

into the pattern I have used or indirectly link with the system.

In summary, I believe the factors presented in the framework cover the key areas of consideration one must tie into a planning and monitoring process.

If managers would use this list in that manner, I believe issues would be identified early enough in the program development cycle so that corrective

actions strategies could be implemented in a timely manner.

Sincerely

,

Robert E. Parsons
NBS»n4A (REV, 2 « 8 C) U.S. DEPT. OF COMM.

BIBLIOGRAPHIC DATA

SHEET (See instructions)

. PUBLICATION OR

REPORT NO.

NBSIR 81-2327

2. Performing Organ. Report NoJ 3.

4. TITLE AND SUBTITLE

Publication Date September 1981

Monitoring System Development: A Framework and Application

5. AUTHOR(S)

Stephen D. Garrity

6. PERFORMING ORGANIZATION (If joint or other than NBS, see in struction s) 7. Contract/Grant No. national bureau of standards

DEPARTMENT OF COMMERCE 8. Type of Report & Period Covered

WASHINGTON, D.C. 20234

F inal

9. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street. City. State, ZIP)

10. SUPPLEMENTARY NOTES

See also related manual: Garrity, S.D. A manual for designing and implementing a process to monitor complex system developments , Washington, D,C.: Experimental

Technology Incentives Program, National Bureau of Standards, September 1981.

i

^ Document describes a computer program; SF-185, FlPS Software Summary, is attached. (NBSIR 81—2328)

11. ABSTRACT (A 200-word or less factual summary of most significant information. If document includes a significant

bi bl iography or literature survey, mention it here)

Projects aimed at developing new systems where there are substantial uncertainties as to system requirements, development processes, and ultimate ownership can present project managers with a range of complex, unstructured problems. A process

to help identify and solve these problems in a timely, controlled manner is of centra]

importance to the successful conduct of a development with these circumstances.

This document describes a research project devoted to the examination of problems

in complex system developments and to the development of a process managers can use tc deal with them. Conducted within the Experimental Technology Incentives Program

(ETIP), the research includes analysis of several ETIP projects, a review of the

systems literature, presentation of a monitoring framework to help manage complex

developments, and a brief application of the framework to one ETIP project.

The proposed monitoring process consists of a framework of thirty factors and a set of five functions which monitoring can serve. The factors are divided into three

main categories — design, process, and user commitment characteristics — which

reflect the general types of problems found to be important in the ETIP environment

and the system literature. The functions of the framework, designed to serve both

administrative and research purposes, include problem identification, strategy development, research, documentation, and dissemination. A model of how the framework

might be implemented within a project is also described.

12. KEY WORDS (Six to twelve entries; alphabetical order; capitalize only proper names; and separate key words by semicolon s) Administrative experiment;

Program; information systems

development

.

13. AVAILABILITY

evaluation systems; Experimental Technology Incentives

; project management; system acquisition; system

14. NO. OF

PRINTED PAGES

Unl imited

^ 1 For Official Distribution. Do Not Release to NTIS 607

'

! Order F'om Suoerintendent of Documents, U.S, Government Printing Office, Washington, D.C. 20402. 15. Price

_X] Order From National i echnical Information Service (NTIS), Springfield, VA. 22161 S41.00

USCOMM-DC 6043-P80