National Aeronautics and

Space Administration

IS20

George C. Marshall Space Flight Center

Huntsville, Alabama 35812
The NASA STI Program...in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA Scientific and Technical Information (STI) 

Program Office plays a key part in helping NASA 

maintain this important role.

The NASA STI Program Office is operated by 

Langley Research Center, the lead center for 

NASA’s scientific and technical information. The 

NASA STI Program Office provides access to 

the NASA STI Database, the largest collection of 

aeronautical and space science STI in the world. 

The Program Office is also NASA’s institutional 

mechanism for disseminating the results of its 

research and development activities. These results 

are published by NASA in the NASA STI Report 

Series, which includes the following report types:

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant 

phase of research that present the results of 

NASA programs and include extensive data 

or theoretical analysis. Includes compilations 

of significant scientific and technical data 

and information deemed to be of continuing 

reference value. NASA’s counterpart of peerreviewed formal professional papers but has less 

stringent limitations on manuscript length and 

extent of graphic presentations.

• TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis.

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees.

• CONFERENCE PUBLICATION. Collected 

papers from scientific and technical conferences, 

symposia, seminars, or other meetings sponsored 

or cosponsored by NASA.

• SPECIAL PUBLICATION. Scientific, technical, 

or historical information from NASA programs, 

projects, and mission, often concerned with 

subjects having substantial public interest.

• TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission.

Specialized services that complement the STI 

Program Office’s diverse offerings include creating 

custom thesauri, building customized databases, 

organizing and publishing research results...even 

providing videos.

For more information about the NASA STI Program 

Office, see the following:

• Access the NASA STI program home page at 

<http://www.sti.nasa.gov>

• E-mail your question via the Internet to 

<help@sti.nasa.gov>

• Fax your question to the NASA STI Help Desk 

at 443 –757–5803

• Phone the NASA STI Help Desk at 

443 –757–5802

• Write to:

NASA STI Help Desk

NASA Center for AeroSpace Information

7115 Standard Drive

Hanover, MD 21076–1320
i

NASA/CR—2012–217471

Elements of Engineering Excellence

J.C. Blair, R.S. Ryan, and L.A. Schutzenhofer

Al Signal Research, Inc., Huntsville, Alabama

November 2012

National Aeronautics and

Space Administration

Marshall Space Flight Center • Huntsville, Alabama 35812

Prepared for Marshall Space Flight Center

under Contract NNM07AA77C
ii

Available from:

NASA Center for AeroSpace Information

7115 Standard Drive

Hanover, MD 21076 –1320

443–757–5802

This report is also available in electronic form at

<https://www2.sti.nasa.gov/login/wt/>
iii

TABLE OF CONTENTS

INTRODUCTION .................................................................................................................. 1

PRINCIPLES OF ENGINEERING EXCELLENCE ............................................................ 1

Root Causes of NASA Failures/Problems ........................................................................... 2

Elements of Engineering Excellence .................................................................................... 4

TECHNICAL UNDERSTANDING AND EXECUTION .................................................... 6

Understanding the Physics .................................................................................................. 6

Ensuring Integration ........................................................................................................... 8

Interactions and Interfaces .................................................................................................. 9

Sensitivities, Uncertainties, and Margins ............................................................................. 11

PARTNERSHIP WITH THE PROJECT................................................................................. 15

Technical Authority ............................................................................................................. 15

Requirements Management ................................................................................................. 16

Risk Management ............................................................................................................... 18

Cooperative Solutions ......................................................................................................... 19

INDIVIDUAL AND ORGANIZATIONAL CULTURE ...................................................... 21

Ownership and Accountability ............................................................................................ 21

Critical Thinking vs. Procedures .......................................................................................... 22

Right People in the Right Places ......................................................................................... 25

Learning Organization ........................................................................................................ 30

SUMMARY ............................................................................................................................ 44

REFERENCES ....................................................................................................................... 46
iv
v

LIST OF FIGURES

 1. Process for Achieving Engineering Excellence ............................................................ 2

 2. Elements of Engineering Excellence ........................................................................... 5

 3. Understanding the Physics ......................................................................................... 7

 4. Successful System Requires Integration of All Aspects .............................................. 8

 5. Problem Produced by Interactions Among Combined Elements ................................ 10

 6. Known Causes of Launch Failures Worldwide by Vehicle Subsystem

(1980–May 2007) ........................................................................................................ 12

 7. Variation in Sensitivity with Performance .................................................................. 12

 8. Progression of Ownership .......................................................................................... 16

 9. Requirements Development Responsibilities .............................................................. 17

10. Coupled Risk ............................................................................................................. 18

11. Risk Taxonomy after DCR ........................................................................................ 19

12. Two Paths of Development ........................................................................................ 27

13. Wheel of Life ............................................................................................................. 28

14. The Forks of the Road Choice ................................................................................... 29

15. Motivations for Choosing .......................................................................................... 30

16. Learning Organization ............................................................................................... 31

17. The Five Learning Disciplines Converging to the Whole: 

The Learning Organization ........................................................................................ 32

18. Characteristics of Mental Models .............................................................................. 33

19. Categories of Advocacy and Inquiry .......................................................................... 34
vi

LIST OF FIGURES (Continued)

20. Shared Vision ............................................................................................................. 36

21. Supports of Commitment .......................................................................................... 37

22. Personal Mastery ....................................................................................................... 38

23. Characteristics of Team Learning .............................................................................. 40

24. Team Vision Alignment .............................................................................................. 40

25. Systems Thinking ....................................................................................................... 41

26. Summary to Success ................................................................................................... 44
vii

LIST OF ACRONYMS, ABBREVIATIONS, AND NOMENCLATURE

CFD Computational Fluid Dynamics

CompStat Computer Statistics

DCR Design Certification Review

HP Horsepower 

IQ Intelligence Quotient 

ISS International Space Station

JSC Johnson Space Center

KSC Kennedy Space Center

LOC Loss of Crew

MIT Massachusetts Institute of Technology

MSFC Marshall Space Flight Center

NASA National Aeronautics and Space Administration

NYPD New York Police Department

PRA Probabilistic Risk Assessment

q, Q Dynamic Pressure

Qα, Qβ Dynamic Pressure Times Angle of Attack or Side Slip Angle

SSME Space Shuttle Main Engine
viii
CONTRACTOR REPORT

ELEMENTS OF ENGINEERING EXCELLENCE

Introduction

The inspiration for this paper originated in discussions with the director of MSFC 

Engineering in 2006 who asked that we investigate the question: “How do you achieve 

excellence in aerospace engineering?” The authors’ approach to answering this question 

was a short course on Excellence in Engineering which is documented in this report. It has 

been taught as an adjunct to the Lessons Learned course at MSFC. The approach, a total 

system approach, forms a triad consisting of: Technical Understanding and Execution, 

Partnership With the Project, and Individual and Organizational Culture. We first looked at the 

“Root Causes” of various failures that NASA has had in its major projects. We found five 

fundamental root causes, which served as a basis for developing the above elements and 

sub-elements of the process. The application of these elements produces a path that leads to 

excellence in the product. We will discuss each of the elements and its sub elements and 

conclude with a short discussion of a Learning Organization as a key element in transforming 

the culture to achieve “Engineering Excellence.”

There are a number of examples and figures cited in this report that are extracted from

the Lessons Learned in Engineering report, NASA CR-2011-216468, and in the Engineering

the System and Technical Integration report, NASA CR-2011-216472.

Principles of Engineering Excellence

o Overview

o Problem Root Causes

o Solutions

The approach used to derive the principles of engineering excellence is shown on 

Figure 1. The process began with a study of the major incidents experienced by the authors 

working for NASA. This study was used to develop root causes from technical, 

organizational and cultural considerations. A path to excellence was developed based on 

Technical Understanding and Execution, Partnership with the Project, and Individual and 

Organizational Culture.
2

Expanding this triad: Technical Understanding and Execution addresses 

understanding the physics, ensuring integration, interactions and interfaces, and sensitivities, 

uncertainties, and margins. Partnership with the Project addresses technical authority, 

requirements management, risk management, and cooperative solutions. Individual and

Organizational Culture addresses ownership and accountability, critical thinking vs. 

procedures, right people in right places, and the learning organization.

Root Causes

Inadequate

Product

• Technical Understanding 

and Execution

• Partnership with Project

• Individual and 

Organizational Culture

Product, Organization & 

Personnel Excellence

* Technical 

* Organizational

* Cultural

Process for Excellence

Figure 1. Process for Achieving Engineering Excellence

Root Causes of NASA Failures/Problems

This study identified five top root causes (others can be added by the reader) that 

have led to major problems in NASA projects. 

1. Shifting from engineering “hands-on” and “excellence” to “insight/oversight”. Lack of 

ownership.

2. “Normalization of the deviances”. Not questioning anomalies.

3. Lack of critical thinking. Over-reliance on procedures and computer codes.

4. Decentralization of authority. 

5. Organizational and technical complexity. 
3

The five root causes are not listed in any priority order. The first root cause listed 

deals with a shift in the NASA culture, where the organization moved from a hands-on 

engineering approach to an insight/oversight approach. In the early days the heritage was 

basically the arsenal approach where you designed, built and verified the system before 

contracting it out for production. The engineers really understood the design, the 

hardware/software and the system based on actual experience. In the early culture much of 

the technology development was an in-house, hands-on activity. The shift has resulted in the 

elimination of much independent analysis and test and experience based understanding of 

the systems required to catch and prevent problems. Howard E. McCurdy in his book Inside

NASA says, “NASA officials from the original cultures believed they needed to provide their 

engineers and scientists with hands-on experience in order to maintain the technical side of 

the house. It was the only way to keep them technically sharp. By keeping their own 

engineers and scientists sharp, they could penetrate the work of the contractor. ... During the 

first decade of space flight, a strong technical culture guided the work of NASA employees. 

The norms typical of that period required NASA to maintain a corps of professional 

employees deeply involved in the details of space flight and aeronautics. The technical 

culture counterbalanced many organizational forces that rose up to challenge it. It 

overpowered the usual bureaucratic tendencies present in government operations. It 

provided a counterweight to the centralizing and organizational necessities of the Apollo 

mission.” [McCurdy, H. 1993] The loss of the technical excellence based on hands-on 

experience has led to many of the problems and therefore is one of the root causes of 

problems. To prevent problems, NASA needs to re-establish the culture of technical 

excellence based on hands-on work.

The next root cause is the normalization of deviances; not questioning anomalies. As 

was evidenced in the Challenger failure, we see deviations in the characteristics that are not 

quite normal, but seem to have no major consequence. After seeing these deviations a few 

times we accept them as normal and ignore them. [Rogers Commission Report, 1986] The 

result is a major failure where the deviation becomes catastrophic. The technical mind must 

question each and every deviation and develop risk-based understanding for operations to 

proceed. The history of space flight is replete with examples such as Challenger (in general 

smaller consequences) of small deviations that have over time become major incidents. 

More will be discussed on this subject later.

The third root cause is closely related to the second in that it is the lack of critical 

thinking with an over reliance on computers, processes and procedures. Computers, 

processes and procedures are necessary but can never take the place of the human mind. 

In the end all of our resources, tools etc. are there as an aid to the human mind and the 

human creativity and decision making. 

The fourth root cause is an organizational and culture cause where authority was 

decentralized to the extent that there was no real decision making authority and personnel 

did not know where to go to get a decision. This led to a floundering and stifled organization 

that was paralyzed by its inability to get actions.
4

The final root cause is closely related to the fourth and is driven by the technical and 

organizational complexity. Space systems are highly complex and very sensitive to small 

uncertainties. This complexity requires in-depth penetration and understanding where small 

technical glitches result in major problems. The technical complexity leads one to think that 

the technical complexity must be matched with organizational complexity. We know that 

when managing the organizational complexity becomes as large as or larger effort than 

managing the technical, then the product success is in question. Simplicity is the pathway to 

product success both technically and organizationally.

Listed below are five of the major incidents used in this study for developing the root 

causes and the preparation of the process of achieving excellence in engineering. In 

parentheses are shown the root cause numbers that pertain to this incident. There are many 

more that were looked at; however, these are representative of the problem cause and 

problem solution. The reader can add to the list as they desire. There are many books 

written on failures, failure analysis, etc. “Beyond Engineering, How Society Shapes 

Technology” [Poole R., 1997], is one book that deals with the complexity factor and the 

influence it has on failures both technically and organizationally. It should be required reading 

for managers and for engineers.

Examples Supporting Root Causes (Numbers in parentheses indicate associated root 

causes)

1. Space Shuttle Challenger ( 2, 4, 5 ) [Rogers Commission Report, 1986] 

2. Space Shuttle Performance ( 3, 5 ) [Holloway, 1999]

3. Space Shuttle Columbia ( 2, 3, 4, 5 ) [Gehman, et al, 2003]

4. Hubble Telescope Mirror ( 1, 2 ) [Harland, et al, 2005]

5. X-33 Single Stage to Orbit ( 3, 5 ) [David, 2001]

Elements of Engineering Excellence

Taking the incidents studied and the resulting root causes led to the development of 

an approach for achieving excellence in engineering. This approach may be divided into 

three elements* as illustrated in Figure 2. First, Technical Understanding and Execution—it is 

basic that engineering products must be technically correct. Second, Partnership with the 

Project—successful products require a positive, productive relationship between Engineering 

and the Project Office. Third, Individual and Organizational Culture—all activities are 

undergirded by the prevailing culture, which must foster the attitudes and behaviors 

necessary for success in producing and operating our complex systems. 

* The triad is the simplest representation of most solutions. If we can reduce our approach to a triad 

then we have a more easily understood means of solving our problems. 
5

Products

Technical Understanding and Execution

• Understanding the Physics

• Technical Integration / T-Model

• Interactions and Interfaces

• Sensitivity, Uncertainty, Margins

Individual & Organizational Culture

• Ownership and Accountability

• Critical Thinking vs. Procedures

• Right People in Right Places

• Learning Organization

Partnership With Project

• Technical Authority

• Requirements Management

• Risk Management

• Cooperative Solutions

• Ensuring Integration

Figure 2. Elements of Engineering Excellence

Each of these elements of the solution has four major divisions that will be discussed 

in detail in the following sections. 
6

Technical Understanding and Execution

While engineering involves aspects of many fields of endeavor, its core is technical 

work. This essential technical work must be done correctly and comprehensively; therefore 

Technical Understanding and Execution is fundamental to Engineering Excellence.

• Understanding the Physics

• Ensuring Integration

• Interactions and Interfaces

• Sensitivities, Uncertainties, Margins

Technical Understanding and Execution

Four aspects of Technical Understanding and Execution will be addressed:

• Understanding the Physics

• Ensuring Integration

• Interactions and Interfaces

• Sensitivities, Uncertainties, and Margins

Understanding the Physics

It is clear that in-depth technical knowledge and expertise is essential and fundamental 

to the engineering process. This is founded on understanding the physics of the system

(Figure 3). 
7

Understanding the Physics

• Technical experts should

- Step back from the details and take a 

broad view of what is happening 

- Consider the real system vs. the model 

(Don’t eat the menu)

- Use simplified models to help 

understand the phenomena (If you 

can’t explain it in simple terms, you 

don’t understand it)

- Explain the technical concepts and 

significance of the results to the 

leadership 

• In-depth technical knowledge and expertise is essential and fundamental

Figure 3. Understanding the Physics

Technical experts should focus not only on the details of their technical specialty, but 

should step back from the details and take a broader view. Understand the physics of the 

system--consider what is happening with the real system, and don’t become enamored with 

the model. The “model” can be either the analytical model, the simulation, or the mental 

model that is carried in our mind. Recognize that these are just representations of the actual 

system, and are necessarily incomplete (and sometimes misleading).

At the same time, models can be necessary tools for our understanding. Use 

simplified models (e.g., back-of-the-envelope approximations, free-body diagrams, etc.) to 

help understand the physical phenomena. Our early mentors strongly emphasized this. 

They correctly thought “If you can’t explain it in simple terms, you don’t understand it”. As a 

corollary to this idea, when communicating technical issues with management, engineers 

owe managers an explanation of the issues and their significance expressed in simplified 

terms.

How does one acquire an understanding of the physics of the system? In addition to 

using simplified models, take every opportunity to see and touch the real hardware. As initial 

components and systems are built and tested, go see them. Lay hands on them. Visit test 

areas and participate in testing. Visit contactor plants where hardware is being built and KSC 

where hardware is being assembled. If the system you are working on is not yet to the 

hardware stage, search out similar hardware and related facilities. These experiences will be 

invaluable in understanding of what the real system is like and how it performs.
8

Ensuring Integration

Experience has shown that most technical problems are systems problems that occur 

because of interactions, interfaces, and communications breakdown. It is of primary 

importance that products are fully integrated in their design and operations. Complex space 

systems are composed of many parts, entail numerous diverse technical areas, and their life 

cycle involves a wide variety of processes and events. Also, the system must meet multiple 

requirements and constraints in safety, performance, cost and schedule. All of these 

aspects must be integrated into a balanced system, as illustrated notionally on Figure 4. 

Figure 4. Successful System Requires Integration of All Aspects

The success of the system depends on ensuring this complex, multi-faceted 

integration. Ensuring that the needed integration takes place is complicated by the common 

natural tendency for individuals to focus on their part or technical area without adequate 

attention to its interactive effects with the rest of the system. While it is necessary to have 

parts and technical areas correctly executed, it is essential that they be fully integrated with 

the rest of the system. This is the basis of the “T-Model” philosophy that illustrates in-depth 

technical execution with an overarching system perspective. 

Successful integration involves multiple approaches, both formal and informal. Formal 

integration includes official instruments such as interface control documents, and dedicated 

organizations such as integration offices and working groups. Informal integration is the 

ongoing communication among technical areas by way of person-to-person interactions, 

discussions, informal meetings, and so forth. Providing means for formal integration is 
9

necessary, but informal integration is the more powerful in achieving an integrated product, 

and can’t be overemphasized. Pervasive communications is a vital enabler of integration.

Those performing the design need to understand where they fit in the 

compartmentalization/reintegration process and the nature of their interfaces and interactions 

as a part the total system. Integration involves finding the correct balance among system 

requirements and constraints, designing for the total life cycle including manufacturing, 

verification, and operations, and bringing together the design and discipline functions for all of 

the hardware and software constituents so as to produce a successful total system. A 

successful system has acceptable performance and is safe, reliable, timely and affordable.

All must be alert for integration issues and proactive in addressing them, 

Management should organize for integration and create an environment that encourages and 

ensures integration. Integration is everyone’s responsibility. 

Interactions and Interfaces

Our complex space systems are highly interconnected and interactive. What happens 

in one subsystem or area usually affects other subsystems or areas, often in ways that are 

unexpected. Reintegrating compartmentalized parts frequently produces system behavior 

not anticipated during the compartmentalization. One way of describing this is “Things come 

apart linearly, but go back together nonlinearly.” There are interactive phenomena such as 

flutter and pogo that only exist in the system context.

Figure 5 illustrates the concept, where elements may perform satisfactorily by 

themselves (or possibly in simple combinations), but when combined with the other elements 

of the system produce unacceptable interactive effects. 
10

Interactions+between+combined+system+elements+can+produce+

phenomena++not+exhibited+by++individual+elements.

Interaction*

Problem

!OK.

!OK.

!OK.

!OK.

!OK.

!OK.

Element*A

Element*B

Element*C

A

B

C

Figure 5. Problem Produced by Interactions Among Combined Elements

It is necessary to understand and manage Information flow among all the 

compartmentalized parts throughout the design process; however, interfaces alone do not 

identify all the interactions that occur when the parts are combined. As noted above, some 

critical interactions can only be addressed from a system perspective.

How do we deal with these interaction and interface issues? Engineers and managers must:

1. Give much deliberate attention to potential interactions, exploring them at each level of 

integration. Specifically evaluate areas of known interactions through analysis and 

testing. Design-in means of reducing or decoupling the interaction, such as pogo 

accumulators.

2. Look for opportunities to apply combined analyses, i.e., analyses that formulate the 

combined problem from first principles instead of handing off between separate 

analyses. An example might be combined thermal-structural analysis of ablative 

materials. Similarly, test with as many combined effects and environments as 

possible.

3. Manage the interfaces and the data flow across them. Know who is responsible for 

each area.

4. Maintain robust informal communications with all disciplines and technical areas. 

“Robust” here means communicating in multiple ways—don’t have single-point failures 

in the communication chain. As stated previously, formal methods of communication 
11

are necessary, but informal communications are more powerful in achieving technical 

integration.

Just as our hardware/software systems are complex, our organizations typically are 

also complex. Organizational interactions and communications can be as complex as the 

technical and require a high level of attention. Organizational structures should not be overly 

complicated, and should streamline information flow as much as possible. It has been said 

that if integrating the organization requires more effort than integrating the technical product, 

clearly something is wrong with the organization. 

Robert Pool [Pool, 1997] has pointed out the need for adaptability in organizations: 

“Layered organizational structure seems to be basic to the effectiveness of 

organizations...some groups are bureaucratic and hierarchical, others professional and 

collegial; others are emergency response. ... Because of complexity, they are best 

decentralized; because of tight coupling, they are best centralized.” Organizations must be 

flexible to adapt to the given situation. “High reliability organizations also emphasize active 

learning, not simply the memorization of procedures.”

Sensitivities, Uncertainties, and Margins

An important aspect of technical understanding and execution of design for high power 

density launch vehicles is associated with sensitivities, uncertainties, and margins. These 

quantities enable the designer to achieve the best design concepts because they provide 

important interactive insights. 

Launch vehicle design goes back to the late 1950’s where there was a lack of 

understanding of environments; material and manufacturing characteristics; computational, 

simulation, and testing capabilities; and overall design experience. During the time frame of 

1957 until 2010 there were 344 major launch vehicle failures [Encyclopedia Astronautica, 

2010]. About 28% of the failures had unknown causes (Russian failure causes during the 

50’s and 60’s are not in public domain). The total percentage failures since 1980 for each 

country are: [Russia, 4.2%]; [USA, 6.1%]; [Europe, 5.8%]; [Japan, 9.4%]; [China, 11.6%]; 

[India, 40%]; and [Rest of World; 50%] [Demidovich, 2007]. The cost of these failures is in the 

billions of dollars and loss of international prestige. 

From 1990 to 2006 Launch vehicle failures have been attributed to engineering 

mistakes, technology surprises, or a combination of both [U. S. Government Satellite ----

2007]. Of the total of 22 failures in that time frame, 7 have been associated with engineering 

mistakes, 10 associated technology surprises, and 5 with a combination of both. The major 

causes of failure by subsystem from 1980 to May 2007 are shown in figure 6 below. 

Historically, propulsion and guidance and navigation systems have been the first and second 

largest causes of failure.
12

 Figure 6. Known Causes of Launch Failures Worldwide by Vehicle Subsystem

 (1980-May 2007)

As the design proceeds, the designer tries to mitigate sources of failure. There are a 

number of ways to accomplish that: catch errors, examine hardware, question modeling and 

testing results, and so on. However, it turns out that determining and understanding

sensitivities, uncertainties, and margins provide necessary insights into a design along with 

insights into all the interactive aspects of multi-subsystem coupling associated with high 

power density launch vehicle systems. 

Sensitivity factors usually pertain to the change of one variable with respect to a 

change in another variable (partial derivative). High performance systems usually have high 

sensitivities. This is illustrated in Figure 7 where the variation of sensitivity and performance 

can be seen. This illustrated variation is typical and, furthermore, high performance systems 

are usually on the steep part of the curve because high performance demands are necessary 

to meet mission requirements.

Figure 7. Variation in Sensitivity with Performance
13

An example of sensitivity with performance can been seen in a comparison of the 

power densities (horsepower / pound of engine weight) of automobile engines, jet engines, 

and rocket engines for various power levels. Their power densities are: (car, .54; Indy engine, 

2.91; small jet, 18.3; large jet, 149.3; rocket engine, 879). The higher the performance a 

system has the higher the sensitivity. If an automobile engine could be designed with the 

efficiency of a rocket engine, its weight would be about 1⁄4 pound. The specific impulse 

efficiency achieved by the current design of the SSME is 98%. In comparing these propulsion

systems, it should be noted that system complexity also increases as performance increases.

Launch vehicles that put payloads in orbit are high performance systems. This can be 

illustrated through the following three comparisons. Firstly, if a comparison is made between 

horsepower per pound of an automobile, commercial airplane, and a rocket stage, it would 

show that the horsepower per pound of a rocket stage is about 2 orders of magnitude higher 

than a commercial airplane and 3 orders of magnitude higher than an automobile. Secondly, 

if a comparison is made regarding propellant mass fraction (ratio of mass of propellant to total 

mass of system) of an automobile, commercial airplane, and a rocket stage, they would be 

respectively about .04, .4 and .9. This means that a rocket stage is 90% propellant and 10% 

structure + other parts; where an automobile is 4% fuel and 96% structure + other parts. 

Finally, if a comparison is made between the thrust required to fly a man on 115 mile 

suborbital flight to an orbital flight to 160 nautical miles, it would take an order of magnitude 

more thrust to go to orbit, e.g. about 800,000 to go to orbit compared to 80,000 pounds of 

thrust for a suborbital flight. These sensitivity comparisons illustrate the differences in the 

various transportation systems and the extreme levels required to be achieved by high 

performance rocket systems. 

Sensitivity factors (partial derivatives) are determined through analysis, test, or 

simulations and can be applied in at least three different ways in design and development. In 

the first, they can be applied to achieve a best balanced design when trying to tune among a 

number of design variables. In the second, when there is hardware in test, sensitivity analysis 

can be applied to confidently make block changes to fine tune operations. In the third, in 

situations where there are high uncertainties in input variables, reducing associated 

sensitivity factors will reduce the uncertainties in the output variables. 

Uncertainties can be determined through historical data bases, test, and expert 

opinion. Prudent designers identify uncertainties and account for them in nearly all aspects 

of design and development. For example, in rocket flight, aerodynamic bending loads are 

proportional to the dynamic pressure (q) multiplied by angle of attack (α). For a circularly 

cylindrical rocket, if there were no wind and no maneuvering of the vehicle, then the angle of 

attack would be zero and thus no bending loads due to aerodynamics. However, there are 

nearly always random atmospheric winds and values of (qα) can be typically between 0 to 

6000 psf-degrees. Because of this uncertainty in the wind, the designer designs for a 3 

sigma wind with a 50 percent confidence level. Then the flight 3 sigma performance 

characteristics (qα) are determined through Monte Carlo simulations [Hanson, J.M. et. al., 

2010]. Quantifying and understanding uncertainty enables risk assessment which provides 

confidence in the designed product.
14

Margins are needed in the design process to provide for unknown uncertainties, 

unforeseen design anomalies, immaturity in the knowledge base, etc. Margins represent the 

difference between an allocated limit and the actual design values including the effects of 

uncertainty. Typically designers are concerned with margins associated with weight, 

electrical power, thrust, specific impulse, etc. They are determined based on historical data, 

maturity of technology, and project phase. Margin tracking provides the project information 

regarding maturity development where it is expected that the margins will reduce as the 

project evolves.

If the designer doesn’t have a clear understanding of sensitivities, uncertainties, and 

margins, then the design cannot be considered to be robust and results in high risk

operations. A failed launch could cost in the neighborhood of about $500 million or more. 

However, clear understanding of these quantities provides confidence in the design, enables 

the project to determine the risk with a high level of confidence, and a successful launch 

enhances our international prestige. 
15

Partnership with the Project

The second element of the Engineering Excellence triad is Partnership with the 

Project. Engineering and the Project Office have distinct but interlocking roles that must 

relate in a positive, productive partnership to achieve successful products. This section will 

address productive partnering of the two organizations and will discuss primary areas of 

cooperation between the organizations. 

Four topics will be addressed:

• Technical Authority

• Requirements Management

• Risk Management

• Cooperative Solutions

Technical Authority

There are various ways Engineering and the Project Office can relate, and past 

experience has seen the pendulum swing between the extremes of having Engineering fully 

involved to essentially having “hands off”. Arguably the most productive relationship results 

when Engineering acts as the Technical Authority for the project, serving as the technical 

conscience that ensures the technical correctness of the product.

What is implied by Technical Authority? It means taking ownership and accountability 

of the technical performance of the product. Ownership produces accountability. There is a 

progression of ownership as illustrated in Figure 8, which takes one from Observer to 

Participant to Owner. The Observer is a spectator who may critique, but still remains 

minimally engaged. Next, a Participant has a “doing” role with shared responsibility. Finally, 

the Owner not only is a doer, but has the full responsibility of ownership. As one moves 

along the progression, there are major increases not only in effectiveness and accountability, 

but also in enthusiasm for the job at hand.
16

Observer Participant OWNER

Spectator

Critiquer

Doer

Shared Responsibility

Doer

Sole Responsibility

Figure 8. Progression of Ownership

“Automatic Responsibility” was a term used in past years to describe Engineering’s 

ownership and accountability role for the technical aspects in all Center projects, either inhouse or contracted. It is most productive when this ownership role is internalized (“felt”) and 

asserted by Engineering, and is understood and relied on by the Project Offices, forming the 

basis for technically sound products

While this approach doesn’t eliminate all conflict between the two organizations, what 

remains is a creative tension that drives toward the best balanced design.

Requirements Management

A primary area of partnership between Engineering and the Project is requirements 

development and management. See Figure 9. 
17

Requirements Development

User

Needs

Performance

Specification

Design

Specification

Top-Level

Mission

Requirements

Architecture

and

Concepts

Derived

Requirements

Technical

Standards

Project Engineering

After development, requirements are formally levied by the Project

Iteration

Figure 9. Requirements Development Responsibilities

The Project organization holds and formally imposes the project requirements, to 

which Engineering designs the product. However, this is not a one-directional flow. Consider 

the sources of requirements. The customer states top-level mission requirements such as 

what mass is to be delivered to target orbits, delivery schedules, cost limits, etc. The majority 

of the remaining requirements are initially generated by Engineering, including architectures 

and concepts, derived requirements on systems and subsystems, and technical standards. 

These requirements are jointly iterated as necessary with the Project, which then formally 

imposes them. So requirements development is a partnership activity. Likewise, subsequent 

requirements management including flow-down allocation, change management, and 

verification planning entails partnering between Engineering and the Project.

Unrealistic and technically uninformed requirements have been sources of major 

problems. Engineers should demand to understand the requirements and push back on any 

that are unreasonable. On the other hand, engineers should resist the tendency to “overengineer” the system beyond its practical requirement. Use judgment on when enough is 

enough—if three decimal places are sufficient, don’t insist on six. Over-engineering not only 

is wasteful of resources, but can obscure insight and understanding. Better can be the 

enemy of good.
18

Risk Management

Engineering in conjunction with Safety and Mission Assurance work together as 

partners with the project to define, accept, and manage risk. Determining and understanding 

technical, cost, and schedule risk throughout the project cycle provides a means to choose

the best design, focus resources, and provide confidence in the designed product. The main 

focuses of technical risk are safety(personnel, assets, and environmental) and 

performance(requirements, operations, and supportability) .

Technical, cost, and schedule risk are inherently coupled. As shown in Figure 10

programmatic risk is constrained by performance requirements, budget, and schedule. 

Furthermore, a change in any of the risks can impact the other two and must therefore be 

assessed.

Figure 10. Coupled Risk

Engineering manages, accepts, and owns risk jointly with the project, unless risk is 

technically unacceptable to engineering. An example of being unacceptable to engineering 

occurred on the second flight after the Columbia incident. On the first flight after Columbia, 

debris was liberated from the External Tank. The proposed fixes on the 34 ice ramps were 

not implemented after that flight. Prior to the second flight, the safety panels from MSFC, 

JSC, and KSC rated the risk the highest and recommended not to fly until the fixes were 

implemented. In addition, NASA’s safety and chief engineers recommended grounding the 

fleet until the fixes were implemented. The NASA administrator overruled those decisions 

reasoning any danger would come during reentry. If a problem occurred during flight, the 

astronauts could make repairs before returning, or wait at ISS for rescue by another Shuttle 

or a Russian Soyuz. The flight flew as scheduled and without incident, see reference, 

[Griffin, 2006].
19

Figure 11 provides a typical risk taxonomy after the Design Certification Review 

(DCR).

Figure 11. Risk Taxonomy after DCR

By the end of design the risk associated with the risk matrix method should be 

minimized to an acceptable level as indicated in Figure 11. Furthermore the risk associated 

with PRA should also be reduced along with the uncertainty.

As can be observed, risk assessment and management are necessary in projects 

where there are high power densities and complexities. It enables the best design decisions: 

e.g., picking configurations, and focusing resources. It also provides confidence in the design 

and operations. Determining risk throughout the design process is mandatory; however

complex it may be. As the design progresses, knowledge of the system and various 

subsystems matures and the risk value and associated uncertainty decreases. Risk 

assessment is determined in a consistent and conservative fashion. 

Cooperative Solutions

Engineering interacts with the project by cooperating with them in providing 

engineering solutions in two ways. In the first, engineering provides design solutions that are 

balanced and meet the requirements and constraints. These solutions are the result of trade 

studies where technical, cost, and schedule are proactively considered. An overview of the 

trade studies is presented to the project and the rationale for selecting the final design is 
20

delineated. Then the selected design is explained in detail along with the associated risk 

assessment. In the second, engineering provides solutions explaining complicated physical 

phenomena. Again, engineering must be proactive in regard to explaining complex 

phenomena in a simplified fashion to demonstrate fundamental understanding and to 

communicate their knowledge to a broader cross-section of the engineering community.
21

Individual and Organizational Culture

The third element of the Engineering Excellence triad is Individual and Organizational 

Culture. Culture is the set of behaviors and beliefs that form the basis of our individual and 

collective actions. It can be a source of our failures, or can be the foundation that undergirds 

our successes. Product success requires creating and nurturing the individual and 

organizational culture necessary for Engineering Excellence.

Four of the many elements of successful culture will be addressed:

• Ownership and Accountability

• Critical Thinking vs. Procedures

• Right People in Right Places

• Learning Organization

Ownership and Accountability

The merits of Engineering taking ownership and accountability for the technical 

aspects of the projects were discussed previously. Here we will address the culture of 

ownership and accountability. In building this culture, there are roles for the leadership, for 

individuals, and for teams.

First, consider the role of Leadership. The main goals of leadership are (1) producing

a successful product and (2) developing people—teaching, mentoring, etc. In pursuing the

first goal--producing a successful product--it is important that line managers know critical 

issues and ensure that they are being appropriately worked in the organization. Technical 

managers are more than administrators—they have responsibility for the success of the 

product. Secondly, in developing people, leaders should foster and encourage ownership 

and accountability in individuals and in teams. Positive expectation and reinforcement are 

powerful principles in guiding their development. Leaders’ attention to these two essential 

functions should not be displaced by administrative “urgencies”.

Next, consider the role of Individuals. Five characteristics of individuals who take 

ownership are:
22

• Competence – Develop and enhance my skills and capabilities

• Cognizance – Penetrate understanding of the design with critical thinking 

• Commitment – Be dedicated to my task

• Accountability – Be responsible for my product

• Passion – Feel the drive of enthusiasm and ownership

Finally, consider the role of Teams. With proper leadership, a team can function at a 

level higher than the level of its most capable member. High performance and ownership is 

attained in teams where the members hold one another accountable, encourage poor 

performers to improve, and establish respect among team members who are held to the 

same high standards. A well-functioning team quickly identifies potential problems by 

questioning one another’s approaches without hesitation. Team members should be free to 

state their opinions, and to explore and challenge each other’s opinions without feeling 

threatened by the dialogue. This characteristic requires an environment of openness. 

Openness and a non-threatening environment result when team members trust each other.

Trust is built on a foundation of integrity of the team members. Integrity means, among other 

things, consistency between actions and words. Trust will be developed over time when 

team members have Integrity, and Trust will lead to the environment of Openness necessary 

for a well-functioning team. So Openness is based on Trust which is founded in Integrity. 

Open, well-functioning teams promote the ownership and accountability culture.

Well-functioning teams converge to appropriate answers. After all opinions have been 

heard, it is time for a decision. Someone must be in charge. The team leader should provide 

clear focus and make the necessary decisions. Building ownership and accountability 

requires cooperative efforts among Leadership, Individuals, and Teams.

Critical Thinking vs Procedures

Critical thinking is fundamental for the success of individuals and organizations. It has 

been studied and debated for over 2500 years. In fact, it was inspired by the early Greeks 

(Socrates) and evolved through the Middle Ages and Renaissance periods until today. As a 

measure of interest, an Amazon search dealing with critical thinking revealed over 38,000 

citations of books written on the subject. Interest in critical thinking is in almost every 

profession and each has its own tailored definition. However, if these definitions are studied 

there are significant similarities. A straightforward top level definition paraphrased from 

reference [Clayton, 2009] is:

Critical thinking is the ability to understand, to make and carry out informed decisions, 

and to create, invent, and discover by efficiently utilizing a lifetime of knowledge, 

experience, common sense, reasoning, intuition, feelings, confidence, and so on.

The purpose here is to illustrate how critical thinking can be typically applied in the 

design of complex engineering projects. In these projects every participant must believe that 

he/she is a critical thinker and that their inputs and questions are valuable.
23

In the design of complex projects, there are fundamental elements associated with 

critical thinking. Some insights associated with the elements “characteristics of individuals 

and organizations” are described below.

“Insights regarding the individual”

1. Confidence in design requires knowledge and understanding of how things work 

including the physics, kinematics, interfaces, interactions, ...

2. Efficient and effective design means not solely relying on historical procedures, 

processes, codes ... but also thinking through the present application, understanding 

underlying principles and assumptions, and if they do not fit the present needs, tailor 

them for current application.

3. Risk can be reduced by challenging all assumptions associated with analysis, test, and 

simulations; assessing sensitivities and uncertainties of results; and quantifying 

margins.

4. Innovative products can be developed by encouraging “critical thinkers” to ask 

questions and think out of the historical box. 

5. A characteristic of highly motivated individuals is their passion for achieving the best 

balanced design no matter the difficulty; they can’t be discouraged; and their passion 

is contagious.

6. Critical Thinkers are always improving their skills. This includes their 

professional/technical, cultural, personal, and spiritual skills.

In summary, the above list is meant to provide “insights regarding individuals” and not be 

all inclusive. Furthermore, it can be seen that the above are integrated, synergistic, and 

focused on developing the best balanced products. 

Considerations will now focus on insights related to the organization.

“Insights regarding the organization”

1. The vision for all elements of an organization must be clear and focused.

 

For example, during the development of the SSME there were 38 incidents that

cost more than $30 million each. In many of these, flow induced effects were 

driving factors. As a consequence, it was decided to implement computational 

fluid dynamics (CFD) into the design process. The direction from management 

was to “make CFD a design tool.”

2. Technical integrity underlies spirit and motivation. Leaders set example; they 

teach/mentor regarding principles, expectations, discipline, and so on. In addition, they 

inspire the search for truth and best balanced design. 

3. Best balanced designs are achieved by questioning all aspects related to technical, 

cost, and schedule issues. Designers should expect questioning and be prepared to 
24

present design options and associated risks along with their sensitivities and 

uncertainties. 

4. Highly interactive feedback should be promoted at all levels of design to stimulate 

technical integration.

5. Critical Thinkers should be recognized for their contributions to the project.

6. An environment should be established that supports and encourages creativity and 

innovation; eliminate the fear of failure.

7. Project communication, e.g., technical integration, can be achieved via peer 

interactions and formal/informal reviews.

In summary, these in addition to others, are important “insights regarding the 

organization.” They indicate the scope and categories of organizational elements to inspire 

critical thinking.

In addition to fundamental elements related to individuals and organizations, another 

fundamental element of critical thinking pertains to “asking questions”. The focus will be on 

individual and organizational questions to induce critical thinking.

In considering questions related to individuals (or design teams), the major focus will deal 

with technical questions leading to a best balanced design in which the level of risk is as low 

as reasonably possible.

“Individual’s technical questions to induce critical thinking”

(Typical list)

1. When are results needed (what milestones)?

2. What are the requirements?

3. Does the architecture make sense or should it be changed?

 (What can go wrong?)

4. What is the design strategy?

5. What design criteria was applied?

6. How were failure modes determined and what are they?

7. Do attributes meet requirements?

8. What is the maturity of input data?

9. What is the source of input data?

10. What are the underlying physics?

11. What methods were used and how were they benchmarked?

12. What assumptions were made and what are their associated sensitivities?

13. What has been the experience using the methods?

14. How do the results compare to the historical similar cases?

15. What are the sensitivities, uncertainties, and margins?

 (Understand the pedigree of results)

16. How are results being used and what are their impacts?

17. What do the hardware and test data tell us?

18. What are the underlying patterns in the data?

19. How were interactions assessed?
25

20. Have interfaces been adequately evaluated?

21. How were all the results used in the risk assessment?

22. Are risk mitigation approaches adequate?

23. Is there a verification plan and how is verification achieved?

24. Have the results been assessed by peers?

25. Is your intuition satisfied; are you comfortable with the results?

These questions and others enable designers to understand and make the best 

informed decisions to achieve a balanced design where the risk is as low as reasonably 

possible. Asking questions is an important element of critical thinking. This should be 

understood by all critical thinkers, i.e. all designers, and encouraged at all levels of the 

project.

Another category of questions pertains to organizational questions to induce critical 

thinking. In this category, the focus is on what is being done, how it’s being done, what are 

the options, etc. The questions below are from NYPD Compstat activities.

“Organizational questions to induce critical thinking”

1. What are/were we trying to accomplish? (The Objective) 

2. How well did we/you execute the plan? (Administrative)

3. What happened? (Descriptive)

4. Why do we/you think it happened? (Diagnostic)

5. What alternatives are there? (Creative)

6. What do we/you think will happen? (Predictive)

7. What is the best choice? (Evaluative) 

An assessment of these questions indicates that question one pertains mainly to 

“planning the work” and questions two through seven pertain to “working the plan.”

In overall summary, critical thinking is inherent in all aspects of design of complex 

engineering products. The purpose here is to illustrate how some elements of critical thinking 

provide confidence and fidelity in the final product. The intention is not to provide an allinclusive review of critical thinking; but provide enough insights to encourage designers to 

review the literature and include, as needed, other elements of critical thinking into their 

design domain. 

Right People in the Right Places

A principle that is emerging deals with the role of a person’s talents and/or “voice” --

some would use the word “calling”-- in organizational success or efficiency. There are 

several books, three of which are referenced, which deal with the subject. Stephen Covey’s 

“The Eighth Habit”, [Covey, 2004] “Good to Great” by Jim Collins, [Collins, 2001] and “Now, 
26

Discover Your Strengths” by Marcus Buckingham and Donald O. Clifton [Buckingham et. al., 

2001]. Collins says, “He first got the right people on the bus (and the wrong people off the 

bus) and then figured out where to drive it. Look, I don’t really know where we should take 

this bus. But I know this much: If we get the right people on the bus, the right people in the 

right seats, and the wrong people off the bus, then we’ll figure out how to take it someplace 

great.” Getting the right people on the right bus in the right seat is responsibility of both the 

employees and management. The probing question is: How do I get the right person in the 

right seat on the bus? Remember:

• Each person’s talents are enduring and unique. 

• Each person’s greatest room for growth is in the area of his or her greatest 

strength. 

• Find your voice, inspire others to find their voice. 

• Discover your strengths.

• The strength of an organization is in its people, all else are aids to the human 

mind.

• Selection is a joint employee and organizational task.

It is not only important to get the right person on the right seat on the right bus, but each 

individual has a responsibility to find their own voice. The following list was a Boeing ad that 

appeared a few years ago that contained some fundamental questions that a person could 

ask themselves that would help understand what drives or motivates a person. 

Motivation

There is a person inside me who wants to:

! Design a church

! Run for office

! Learn to cook

! Build the Space Station

! Write a song

! Counsel youth

! Climb a mountain

! Live in Spain

–Boeing Ad in Launchspace magazine

What motivates a person is one key to what their strengths are. Other questions are:

What are my interests?

– Family?

– Professionally?

– Socially?

– Avocationally?
27

What do others ask me to do? Expect of me? Reinforce in me?

– Family?

– Professionally?

– Socially?

– Avocationally?

What do I see that needs to be done? Your calling is what you recognize as needs.

What pulls me (calling)?

What do I admire in others? 

Who are my role models?

When Robert Ryan was a student a Peabody College the Head of the Psychology 

department of Vanderbilt University presented the following figure (Figure 12) that illustrates 

the two main paths that an individual can take in his or her development. The choice is 

having versus being. Having is a social emphasis while Being is spiritual in nature. In 

general from a personal and organizational standpoint the selection of being is the obvious 

choice. It has been said by many that if one chooses being and becomes something of 

value, adding meaning to themselves and society, then the having will take care of itself to 

the extent all basic needs are more than met. Having focuses on position, money, authority 

and accolades while Being focuses on values, spiritual dimensions and adding meaning to 

society as well as to oneself. 

Substance

Personal

Being

Success

Social

Having

Focus

Figure 12. Two Paths of Development

Another imperative that the same professor gave is illustrated in Figure 13, called the 

wheel of life. Each individual’s life is composed of spiritual, mental, physical, social and 

emotional components which should be balanced to the degree possible. What happens is 

that an individual concentrates on their strengths and ignores their weakness. The better one 

can balance these areas the smoother life can be, yet our greatest room for growth is in our 
28

areas of strengths. The paradox is obvious in that we grow by working on our strengths, the 

area of greatest value to an organization, yet we must somehow work on weakness in order 

to have a better balanced life. It is called the principle of compensation in that we build on our 

strengths to overshadow our weakness. In doing this our life is increasingly more unbalanced 

and does not run as smooth. In general one cannot get a total balanced set; however, if one 

is aware of the principle and what is taking place, coping is much easier.

Must understand the principle of compensation (overcompensation for weakness), 

balancing out the five areas of life to the degree possible, then accepting the results.

The Wheel of Life

Spiritual Mental

Physical

Emotional

Social

Mental

Spiritual

Physical

Emotional

Social

Integrated Person Typical Person

Figure 13. Wheel of Life

The principle of getting the right people in the right seats on the right bus [Collins, 

2001] dictates that each individual must be where their strengths, talents and, as Covey says, 

their voice directs what they be [Covey, 2004]. A few questions and ways of finding that voice 

have been discussed in the previous paragraphs and are guides to help implement the 

principle. In summary, quoting from Covey’s book, is the final pearl of wisdom:

“Find your own voice; help others find their own voice.” - Stephen Covey

The next overriding principle in the process of getting the right people in the right seat 

on the right bus is the principle of Choice, both individually and organizationally. Figure 14 

shows the principle as coming to the forks of the road. In New Hampshire, where Robert 

Frost once live in a house at the foot of the hills there is a path which leads from the house 

through bushes. The path comes to a fork where one fork in the path leads into the foothills 

and the other leads into a meadow. There is a tree sitting in the point of the fork and on that 

tree is Frost’s poem “The Road Not Taken”. The story is told that as he was heading down 

the path one day and came to the fork of the paths he got the inspiration for the poem. If you 
29

are ever in that region go by the house and walk the path. The message is clear: at any one 

instant you can only take one of the paths of the fork in the road, which is choice. 

Where You Go Depends on the Choice You Make

The Forks of the Road Choice

Choice

Figure 14. The Forks of the Road Choice

There are six motivations which drive how a person makes a choice, as illustrated on 

Figure 15 [Covey 2004], shown in ascending order to the highest level of motivation. The 

lowest way to make a choice is motivated by anger and results in rebellion or quitting. The 

next level of motivation is to make a choice out of fear and is called malicious obedience. 

Next up the list is doing what I do for the reward it gives and is called willing compliance. 

Duty is the next higher motivation and is fundamental in many of things we do such as 

service to country, community etc. It is normally referred to as cheerful cooperation. Love is 

a very high motivational approach to making our choices, associated with heartfelt 

commitment. The highest level of motivation is adding meaning to the organization, the 

individual and society and is associated with creative excitement. It is very important that as 

an organization and as individuals, we work diligently to cluster our motivations for choices in 

the top three and make basically none on anger and very few based on fear. Using meaning, 

love and duty as our primary motivations builds the individual, the organization and the 

culture.
30

FREEDOM AND POWER IN PEOPLE TO CHOOSE

Creative excitement

Heartfelt commitment

Cheerful cooperation

Willing compliance

Malicious obedience

Rebel or quit

MEANING

LOVE

DUTY

REWARD

FEAR

ANGER

Choices

Figure 15. Motivations for Choosing

The organization has a major role in helping individuals find their voice, developing 

their voice and placing them in the organization based on their voice. Of prime importance is 

for the organization to keep a constant focus on recognizing each employee’s unique gifts 

and capabilities. Once management has recognized the individual’s unique gifts and 

capabilities their next task is to put them in the right place in the organization to utilize those 

characteristics. The final responsibility of management is to foster their continued growth. 

Deming in courses several years ago said, “Management should approve any course an 

employee wants to take even if it does not have direct application to their job.” Developing the 

whole person has organizational impact due to the individual’s development. The quote by 

Albrecht should be an underlying principle for any management and organization.

 “ When an employee shows up for work, you’ve already purchased his or her...IQ points, 

or at least you have an option on them. At the end of each day, you have either exercised the 

option or you’ve let it expire. That day will never come again, and the option on that day’s IQ 

points is gone forever.”

- Karl Albrecht, quoted in Faster Learning Organization [Guns, 1996]

Learning Organization

The method by which we implement the principles just discussed for getting the right 

person in the right seat on the right bus and for developing both the organization and the 

individuals in the organization is through “A Learning Organization.” The following banner 
31

(Figure 16) states that the secret of moving into the future is for the organization to become a 

Learning Organization. The question faced is: What is a Learning Organization?

Figure 16. Learning Organization

There are many viewpoints of what constitutes a learning organization. We have 

studied many of the characterizations and selected the Senge model of a learning 

organization to discuss as an example [Senge, 1990, 1994]. The following is a discussion of a 

Senge learning organization model.

Peter Senge and others have looked at the complexity of current organizations and the 

speed that technologies are evolving in our culture. He and others find that the world is a 

world of fragmentation or compartmentalization. We grow up in a world that continually 

teaches us to fragment our world and establish boundaries. This results in teaching us to play 

roles to fit the situations. Compartmentalization eventually leads to losing connection to the 

larger whole and a feeling of loss. Trying to assemble these fragments is like trying to 

assemble the parts of a broken mirror, in which case it is better to get a new mirror. It is

concluded that the only means of survival is that the organization and the individuals become 

learning organisms. The following figures are taken from a course on “Learning 

Organizations” developed by Robert Ryan and Coy Brown that show the essence of Sengeʼs 

“Learning Organization”. The artwork of the figures was done by Coy Brown. 

Quoting Senge: “Today, I believe, five new ʻcomponent technologiesʼ are gradually 

converging to innovate learning organizations. Though developed separately, each will, I 

believe, prove critical to the othersʼ success, just as occurs with any ensemble. Each provides 

a vital dimension in building organizations that can truly ʻlearn,ʼ that can continually enhance 

their capacity to realize their highest aspirations.” He asserts that these emerging and 

converging disciplines work together as a system which if implemented will grow and keep an

organization viable. Figure 17 illustrates the integration of the five disciplines, with systems 

thinking being the fifth discipline and the integrator. 
32

The Five Learning Disciplines

are Gradually Converging 

Today, Five New Component Technologies are Converging to 

“Innovate” Learning Organizations. Figure 17. The Five Learning Disciplines Converging to the Whole:

The Learning Organization

The discussion that follows will be only an introduction to learning organizations. The 

reader is encouraged to explore books from the reference list to dig deeper into the subject. 

We will now discuss the characteristics of each of its five disciplines: mental models, shared 

vision, personal mastery, team learning, and systems thinking.

A. Mental Models

“What you see is what you expect to see” is a truism repeated over and over. Senge 

says, “Our mental models determine what we select and thus the personal and organizational 

growth.” We have all played the party game where a rumor is whispered in someoneʼs ear. 

That person then whispers what they have heard, to the next ear. The process is continued 

until the end of the line is reached. The last person now tells what has been heard. In 

general it nowhere comes close to the original statement. Eyewitnesses to an accident will 

see different things. The problem is that we are not aware of these tendencies to modify our 

perceptions by our mental models. Mental Models are tacit, existing below the level of 

awareness, and therefore they are often untested and unexamined (Figure 18). For 

individualsʼ and organizationsʼ growth these models must be brought to the surface for 

exploration and discussion to see how they (the lens through which we view things) impact 

our lives.
33

Mental models are the “pictures” we carry in our 

heads...the way we look at life and work...our 

personal framework for how we make sense out of 

the world, and how we take action.

Our mental models are deeply ingrained assumptions, 

attitudes, generalizations, pictures, or images that influence 

our behavior.

The skills of Reflection, Inquiry, and Dialogue, which help us surface and 

evaluate our mental models, are essential to effectively analyzing and taking 

action on critical issues.

Very often we are not consciously aware of our personal mental models, or the 

effects they have on our behavior; but, in fact, we are seeing the world through 

our own personal lens.

Mental Models

Figure 18. Characteristics of Mental Models

A concerted effort must be made to find ways to reshape the lens and find ways to 

better focus the true image (truth). To grow then we must change who we are, our 

paradigms, in order to see reality, the truth. Knowledge and information are critical 

commodities of the future. Future success is much more likely when continuous learning 

becomes an integral part of the culture of an organization. [Senge, 1990, 1994] Mental 

models are one of the keys to the learning process.

The pitfalls we have in surfacing and evaluating our mental models are;

1. Our beliefs are the Truth

2. The Truth is obvious

3. Our beliefs are based on real data

4. The data we select are the real data

He goes on to say that the approach taken must balance advocacy with inquiry--that 

we must lay out our reasoning and thinking and encourage others to challenge it. This means 

that the only way to change is make yourself vulnerable. Robert Hargrove lays out problems 

incurred in balancing advocacy and inquiry. [Hargrove, 1995]. This is a very difficult task 

since we all have the tendency to become defensive; therefore we are not open to the 

challenge.

We must develop the skills of how to balance advocacy and inquiry for self 

development, thus organizational development. The only way an individual and thus an 

organization can grow is by changing the individual’s mental models. Otherwise everything is 

heard and interpreted in terms of our mental models. Senge uses the Ladder of Inference as 

the model of how we quickly jump to an interpretation of the situation based on the mental 
34

models. We can change mental models through reading, visuals etc. but is done most 

efficiently through dialogue.

Figure 19 depicts four different top level categories of mental models. The lowest is 

low advocacy and low inquiry where the person is withdrawn and silent who does not have 

any ideas, views or even any questions. This is saddest of all people. For example, when 

teaching high school we had a young lady that never had a question, a thought or any kind of 

statement. She was unresponsive in any adequacy inquiry situation and showed very little 

signs of growth. All the teachers tried to reach her and bring her out but with very little 

success during the time we were teaching there. This is probably the saddest situation we 

have encountered where the personality is not open to advocacy and inquiry. It is very hard 

to lead something that is not moving in some direction. In contrast the person who is always 

only advocating tends to dominate and suppress the others in a group. People that just ask 

questions without any views can also dominate and stagnate a group in achieving learning. 

The best is a balance between advocacy and inquiry, where each states their views and is

then open to inquiry from others. This dialogue results in a change in mental models and is a 

very powerful growth engine. This balance is what every organization and group should 

strive for.

HIGH ADVOCACY / HIGH INQUIRY

* Communicate: Conclusions, data, 

steps in your reasoning

* Inquire to discover steps and

gaps in one another's reasoning

LOW ADVOCACY / HIGH INQUIRY

* Asking questions but not

revealing your views

HIGH ADVOCACY / LOW INQUIRY

*Advocating your view but not 

inquiring into others’ views

LOW ADVOCACY / LOW INQUIRY

* Not revealing your views or

questioning other's views

* Silent withdrawal

Advocacy and Inquiry

Figure 19. Categories of Advocacy and Inquiry
35

Outdated mental models can hinder learning. Many insights into new markets or 

improving outmoded organizational practices fail to get into practice because they conflict 

with powerful, tacit mental models. There are three questions an organization must ask:

1. What are some of the powerful mental models that exist in our organizations 

currently?

2. How are existing mental models affecting the way we practice our business?

3. Should we change any of our mental models?

Remember:

Outdated Mental Models Can Hinder Learning

The following Talmudic teaching sums up the discussion on mental models.

B. Shared Vision

Experts all agree that vision, personally and organizationally, are the foundation of 

growth and excellence. The early prophets stated in the Bible, “Without a vision the people 

perish.” Others have stated the principle in many ways all alluding to the primacy of vision. 

Senge treats shared vision extensively. The following is a short summary of this treatment. 

Enabling organizations to truly become a learning organization and succeed in todayʼs 

world of “Whitewater” there must be a shared vision (Figure 20). The shared vision emerges 

from personal vision. The shared vision implies that both the organization and the individual 

must make adjustments of their visions in order to have a true-shared vision. There are 

various analogies of organizational shared vision, the hologram being the most powerful. As 

in a hologram, if you start with the whole and cut it up in pieces the image does not 

fundamentally change. In reversing the process by adding up (combining) the pieces the 

whole does not fundamentally change. Organizationally the implications of this model are 

clear.
36

• Shared Vision

– Alignment of values

– Empowered people

– Inspired workforce

– Involved and committed leaders

Shared Vision

Figure 20. Shared Vision

The following is a list of characteristics of shared vision.

• Shared visions emerge from personal visions. 

• The hologram is the example. When you add up the pieces, the whole (image) does 

not change fundamentally.

• Requires that you “hold” the vision while remaining committed to seeing current reality 

clearly.

• Involves

– Telling

– Selling

– Testing

– Consulting

– Co-creating

• Must be based on the purpose of the organization. Focused around building shared 

meaning. Collective sense of what is important.

• A shared vision implies commitment.

The shared vision is collective sense of whatʼs important based on the purpose of the 

organization. It involves telling, selling, testing, consulting, and co-creating. Obviously a 

shared vision implies commitment. Commitment has four pillars of support as illustrated on 

the Figure 21 [Kinlaw, 1993]. Clarity is required for commitment. In general one cannot truly 

commit to something not understood. Competence is another key. Each individual must 

bring their skills/competencies to bear in a support to the shared vision. This is a key part of 

commitment as is influence. Appreciation of both the individual and team efforts supplies the 

energy for continued growth and performance.
37

The Supports of Commitment

Dennis Kinlaw

Commitment

C

L

A

R

I

T

Y

C

O

M

P

E

T

E

N

C

E

I

N

F

L

U

E

N

C

E

A

P

P

R

E

C

I

A

T

I

O

N

Figure 21. Supports of Commitment

Clearly shared vision is important to a learning organization. The shared vision 

provides the dynamic tension between the current reality and what we are going to be. The 

shared vision keeps the organization and the individuals focused on what they are becoming. 

C. Personal Mastery

Personal mastery is the thing we appreciate so much in watching performances. For 

example, Michael Jordan, with his exhibition of the skills and mastery of basketball, is 

something to behold. The grace, flow, and ease at which he appears to perform are a 

testimony to his dedication and practice in obtaining the mastery he displays. But the big 

element of personal mastery that led to the development of all these skills was his ability to 

master himself and dedicate his energies to mastering these skills. The same can be said of 

any individual who masters his/her field, or the performance of a group such as the Boston 

Pops or the Philadelphia Symphony. The orchestra is a good model to illustrate both personal 

mastery and shared vision. The shared vision is the production of a great composition, 

composed by someone renowned. The leader or someone else may rearrange the original to 

produce a different production (sound) and display some of the individual skills of the 

members. The commitment to the arrangement is the shared vision. Personal mastery is the

key: however; each individual member must play their part to near perfection or the clear 

harmonious sound is not present. In addition, regardless of the personal mastery achieved, 

the individual cannot dominate the presentation or again the blend is destroyed. The beauty 

is the blending between the strings, the winds, the percussion, etc. A second factor comes 

into play when someone achieves such personal mastery. They raise the level of 

performance of everyone else on the team, the orchestra, and the organization. 
38

The story is told that when Knute Rockne had one of his great teams at Notre Dame,

the backfield (The Four Horsemen) was outstanding. They became arrogant and cocky and 

looked for all the praise, downplaying the efforts of the other team members. In a critical 

game where Notre Dame was moving the ball down the field with ease, Rockne removed the 

first string lineman, substituting the second string. All of a sudden the backs were being 

tackled in the backfield. They could not move the ball down the field. Rockne let the situation 

repeat for awhile before inserting the first string lineman back into the game. The story goes 

that Rockne never had any more trouble with the Four Horsemen again. Personal mastery 

committed to another vision or goal than the one the team is playing always results in poor 

team performance. Personal mastery committed to the shared vision results in high 

performance. 

Commitment to a vision without personal mastery also results in poor performance. 

Personal mastery is fundamental to success of a team and a learning organization. Personal 

mastery is the thing most obvious and inspiring of all the characteristics of learning 

organizations. Senge deals extensively with the characteristics in the references and in 

Figure 22 which summarizes some of his points.

Personal Mastery is the discipline of 

continually clarifying and deepening our 

personal vision, of focusing our energies, 

of developing patience, and of learning 

how to see current reality more clearly. It 

entails a level of proficiency and involves 

aspiration. 

Personal Mastery

Figure 22. Personal Mastery

As evidenced by the examples given, organizations learn only through individuals who 

learn. Individual learning/mastery is based on clear principles and is grounded in 

competence and skills but goes beyond. “It means approaching life as a creative work, living 

life from a creative as opposed to a reactive viewpoint.” [Senge, 1990, 1994] As Senge and 

others have so clearly stated, the development process is fueled by a dynamic tension 

between the personal vision and reality, which is the source of the creative energy.

“The essence of personal mastery then is how to generate and sustain creative 

tension in our lives. This is anchored by personal vision, which focuses on ultimate intrinsic 
39

desires and not secondary goals.” [Senge, 1990, 1994] Visions must be in alignment with his 

or her purposes--a sense of why he or she is alive. Based on a vision aligned to purpose, 

meaningful goals can be developed. Senge and others emphasize that personal mastery is 

the integration of reason and intuition. The role of emotional intelligence has been dealt with 

in [Weisinger, 1998] and by many others, detailing its importance in growth. Personal 

mastery is always a commitment to the whole discussed previously and teaches choice. 

Choice is fundamental in all of life processes. Each person is unique and has a special task, 

a calling, to fulfill; if not developed and performed it cannot be replaced. 

In the end the prime characteristics of personal mastery are the control and focusing of 

their energy on the vision of their being, growth, and its application to the organizational 

vision and the current task.

D. Team Learning

Much has been written on teams and teaming, for example in references [Martin, Don, 

1993] and [Rich, B. 1994]. It is the in-vogue thing to do today--everyone has teams. 

Organizationally, learning must be both individual and collective as illustrated previously. 

Senge has said, “Collectively we can be more insightful, more intelligent than we can possibly 

be individually. The IQ of the team can potentially be much higher than the IQ of the 

individuals.”[Senge, P. 1990, 1994]. The teaming idea is not new however. In the Old 

Testament, Ecclesiastics 4; 9-12, says, “Two are better than one; because they have a good 

reward for their labor. For if they fall, the one will lift up his fellow: but woe to him that is alone 

when he falleth: for he hath not another to help him up. Again, if two lie together, then they 

have heat: but how can one be warm alone. And if one prevail against him, two shall 

withstand him; and a threefold cord is not quickly broken.” If one is open to the exchange, the 

vulnerability, growth is essentially unlimited. Clearly certain functions can only be performed 

as a team. Basketball, the symphony, the design and building of a high performance aircraft 

are a few examples of teams. Figure 23 outlines the characteristics of team learning.

Team learning is the process of aligning and developing the capacity of a team to create 

the results its members truly desire. It is a discipline that requires a commitment to 

continuous study of the interpersonal group dynamics and the team problem-solving tools. 

The discipline includes becoming skilled in slowing down our thinking processes and 

observing and analyzing the group “process”—becoming aware of ways to enhance clear 

communication. Figure 24 illustrates the power of alignment, as does the laser beam.
40

Team Learning is Vital for Organizational Health. When Teams are Truly 

Learning, They are Producing Extraordinary Results and the Individual 

Members are Growing More Rapidly Than Could Have Occurred 

Otherwise. 

“Dialogue” is the Capacity of 

Members of a Team to Suspend 

Assumptions and Enter Into a 

Genuine “Thinking Together” Allowing 

the Group to Discover Insights Not 

Attainable Individually.

Team Learning

Figure 23. Characteristics of Team Learning

Team Vision

(Alignment)

Individual Vision

(Unfocused)

Figure 24. Team Vision Alignment

Team learning is characterized by dialogue that causes one to become observer of 

their own thinking, self-mastery/self knowledge that involves looking to knowledge of, 

alignment with others on the team, suspends assumptions in order to honor the position of 
41

other viewpoints, is a trusting relationship, and emphasizes diversity with unity. [Senge, P. 

1990, 1994]. Team learning requires a dedication, commitment, sacrifice, respect, 

appreciation, and alignment. When these are present, learning and accomplishment are

unprecedented. The focusing of the human minds as teams on a vision and goal (solving 

crippling problems) heightens the receptivity, creativity, and insight of each individual on the 

team. The power/leverage in organizational and personal life is in the balancing between 

conflicting requirements, or said differently, dealing with the paradoxes of life. 

The following is a set of characteristics of team learning.

• “Collectively, we can be more insightful, more intelligent than we can possibly be 

individually. The IQ of the team can, potentially, be much greater than the IQ of the 

individuals.”—Senge 

• Requires dialogue. In dialogue, people become observers of their own thinking.

• It starts with self-mastery/self-knowledge, but involves looking outward to knowledge 

of, and alignment with, others on your team. 

• Suspends assumptions to honor the passion of the other personʼs viewpoint.

• It is a trusting relationship.

• Emphasizes diversity with unity

• "Alignment of each to achieve a common goal.

E. Systems Thinking

The fifth element of Senge’s Learning Organization is Systems Thinking. [Senge,

1990, 1994] (Figure 25) The scope and impact of systems thinking is so broad and 

important, that the reader is encouraged to study it in depth. System thinking is a key to 

many elements related to organizations, individuals, engineering, etc. It takes many forms. 

It has been our experience that 80% of the time the root cause of engineering problems and 

failures is not in various disciplines but is a breakdown in systems/technical integration. 

System Thinking:

Parts of the Whole System Influence 

all the Other Parts, and That 

Influence is Often/Usually Hidden 

From View. (You Gotta Look for it!)

Figure 25. Systems Thinking
42

Senge provides some of the characteristics of systems:

• Structure Influences Behavior

- "The System causes its own behavior"

- When placed in the same system, people, however 

different, tend to produce similar results. 

• Structure in Human Systems is Subtle

• Leverage Often Comes from New Ways of Thinking

! Systemic Structure (Generative)

! Patterns of Behavior (Responsive)

! Events (Reactive)

• Business and Human Endeavors are Systems, Bound by Invisible Fabrics 

of Interrelated Actions.

• Systems Thinking is a Conceptual Framework, a Body of Knowledge and 

Tools That has Been Developed Over the Past 50 Years, to Make Full 

Patterns Clearer, and to Help Us see how to Change Them Effectively.

Essentially system thinking pictures a process with every element/parameter a cause 

and effect. Therefore system thinking starts with the concept of feedback. The feedback 

can exist as reinforcing loops (engine of growth and instability) and balancing loops 

(stabilizing loops). [Senge, 1990, 1994].

It is very important to understand, appreciate and balance between the reinforcing loop 

and the balancing loop. Senge gives many examples of these two loops. Key to this 

understanding is the inherent characteristics of delays or lags. 

The nemesis to system thinking is compartmentalization, fractionalization, and 

specialization. Although it is necessary to have compartmentalization, fractionalization, and 

specialization., they must always have a system focus or viewpoint or things like the tragedy 

of the commons will occur. [Senge, 1990, 1994]

System thinking then is the coordinator and integrator of the system that makes it play 

together as a whole, yet is a fundamental part of each of the other disciplines and influences 

what they are. This is a very condensed treatment of learning organizations. 

Additional insight can be gained from the 9 Principles and 27 Lessons we developed in 

a short course: “Lessons Learned in Engineering” and a NASA CR report [NASA

CR-2011-216468] that are summarized below.

 I. System success depends on the creativity, judgment, and 

decision-making skills of the people

1. People Are the Prime Resource for Project Success

2. People Skills are Mandatory for Achieving Successful Products
43

II. Space systems are challenging, high performance systems

3. Demand for High Performance Leads to High Power Densities 

and High Sensitivities

III. Everything acts as a system (whole)

4. Systems and Technical Integration

5. Risk Management

6. All Design is a Paradox, a Balancing Act

IV. The system is governed by the laws of physics

7. Physics of the Problems Reigns Supreme

8. Engineering is a Logical Thought Process 

9. Mathematics Is The Same!

10. Fundamentals of Launch Vehicle Design

V. Robust design is based on our understanding of sensitivities,

uncertainties, and margins

11. Robustness

12. Understanding Sensitivities and Uncertainties is Mandatory

13. Margins Must Be Adequate

VI. Project success is determined by life cycle considerations

14. Design Space Constrained by Where You Are in the Life Cycle

15. Concept Selection and Design Process

16. Requirements Drive the Design

17. Designing for the –ilities and Cost

VII. Testing and verification have an essential role in development 

18. Hardware and Data Have the Answers

19. Can Test Now or Will Test Later

20. Independent Analysis, Test, and Design are Keys to Success

21. All Analyses and Tests are Limited

22. Scaling is a Major Issue

VIII. Anticipating and surfacing problems must be encouraged

23. Must Hear and Understand All Technical and Programmatic Opinions

24. There are No Small Changes!

25. Expect the Unexpected

IX. Leadership is the foundation

26. Integrity 

27. Focus Beyond Yourself
44

Summary

In summary, we have looked at the process whereby excellence in engineering can be 

achieved in a project and an organization. Directors of Engineering at Marshall Space Flight 

Center have added the core values that an organization must have in achieving the goal of 

successful products:

Have Integrity

Internalize Teamwork

Be Safe

Communicate

Engage

Make Yourself Better

Leave It Better

Do It Now

The following figure (Figure 26) depicts the process of achieving excellence in 

engineering, starting with understanding root causes of problems, instituting the triangle of

Technical Understanding and Excellence, Partnership with the Project, and Individual and 

Organizational Culture. Product success depends on the application of all the principles of 

engineering excellence.

Figure 26. Summary to Success
45

In closing there is an attitude that must permeate the organization and the individuals 

that can be classified as servanthood. The poem The Bridge Builder by Will Allen Dromgoole

sums up this attitude.

The Bridge Builder

An old man, going a lone highway,

Came, at the evening, cold and gray,

To a chasm, vast, and deep, and wide,

Through which was flowing a sullen tide.

The old man crossed in the twilight dim;

The sullen stream had no fears for him;

But he turned, when safe on the other side,

And built a bridge to span the tide.

“Old man,” said a fellow pilgrim, near,

“You are wasting your strength with building here;

Your journey will end with the ending day;

You never again must pass this way;

You have crossed the chasm, deep, and wideWhy build you this bridge at evening tide?”

The builder lifted his old gray head;

“Good friend, in the path I have come,” he said,

“There followeth after me today

A youth, whose feet must pass this way.

This chasm, that has been naught to me,

To that fair-haired youth may a pitfall be.

He, too, must cross in the twilight dim;

Good friend, I am building the bridge for him.”

- Will Allen Dromgoole
46

References

Blair, J.C., Ryan, R.S., Schutzenhofer, L.A., and Humphries, W.R., “Launch Vehicle Design

Process: Characterization, Technical Integration, and Lessons Learned,” NASA TP-2001-

210992, 2001.

Blair, J.C., Ryan, R.S., and Schutzenhofer, L.A., “Lessons Learned in Engineering,” NASA

CR-2011-216468, 2011.

Blair, J.C., Ryan, R.S., and Schutzenhofer, L.A., “Engineering the System and Technical 

Integration,” NASA CR-2011-216472, 2011.

Buckingham, Marcus, and Clifton, Donald, “Now Discover Your Strength” The Free Press,

New York, 2001.

Clayton, C., http://www.chuckclayton.com/articles/Critical_thinking_definition.htm, January, 

2009.

Collins, Jim. “Good to Great” Harper Business, New York, 2001.

Covey, Stephen R., “The Eighth Habit; From Effectiveness to Greatness,” Free Press, New 

York, 2004.

David, L., NASA Shuts Down X-33 and X-34, March 1, 2001.

http://www.space.com/missionlaunches/missions/x33_cancel_010301.html

http://www.space.com/businesstechnology/technology/x33_newlease_001002.html

Demidovich, Nickolas, Launch Vehicle Failure Mode Database, AST Research and 

Development Project, COMSTAC RLV Working Group, May 17, 2007.

www.faa.gov/.../Failures%20RD%20study-COMSTAC%20RLVWG%20May%202007-1A.ppt

Deming, E., “My Personal Notes from Edward Deming, Class 1990.” Newport Beach, CA.

Encyclopedia Astronautica, The Wrong Stuff - A Catalogue of Launch Vehicle Failures, 2010.

http://www.astronautix.com/articles/thelures.htm

Griffin, M. March, 19, 2006. 

http://www.guardian.co.uk/science/2006/jun/19/spaceexploration.usnews

Guns, Bob, “The Faster Learning Organization,” Jossey-Bass Inc., San Francisco, CA, 1996. 

Gehman, H. W., Columbia Accident Investigation Board, August, 2003.

http://caib.nasa.gov/news/report/pdf/vol1/full/caib_report_volume1.pdf

Hanson, J.M. and Beard B.B., Applying Monte Carlo Simulation to Launch Vehicle Design and 

Requirements Analysis, NASA/TP-2010-216447, September 2010.
47

Harland, David M. Lorenz, Ralph D., Space Systems Failures, p 328-329, Praxis Publishing, 

2005.

Hargrove, Robert, “Masterful Coaching” Jossey-Bass Inc., San Francisco, CA, 1995.

Holloway, T. W., Manager, Space Shuttle Program, Shuttle System Design Criteria, Vol. 1, 

Shuttle Performance Assessment Databook, Revision B, March 16, 1999.

http://pbma.nasa.gov/docs/public/pbma/bestpractices/bp_jsc_49a.pdf

http://pbma.nasa.gov/docs/public/pbma/bestpractices/bp_jsc_49b.pdf

http://pbma.nasa.gov/docs/public/pbma/bestpractices/bp_jsc_49c.pdf

http://pbma.nasa.gov/docs/public/pbma/bestpractices/bp_jsc_49d.pdf

Kinlaw, Dennis C., “Coaching for Commitment”, Pfeiffer & Company, San Diego, CA, 1993

Martin, Don, “Team Think” Penguin Books USA Inc., London, England 1993.

McCurdy, Howard E., “Inside NASA”, The Johns Hopkins University Press 1993.

NYPD Compstat, Personal communication with New York Police Department.

Pool, Robert, "Beyond Engineering, How Society Shapes Technology", Oxford University 

Press, Oxford, New York 1997.

Rich, Ben R., “Skunk Works”, Little Brown and Company, New York, NY, 1994.

Rogers, W. P., Report of the Presidential Commission on the Space Shuttle Challenger Accident, 

June 6, 1986. http://history.nasa.gov/rogersrep/genindex.htm

Senge, Peter M.: “The Fifth Discipline” Bantam Doubleday Dell Publishing Co., New York,

New York, 1990 & 1994.

Senge, Peter M.: “The Fifth Discipline Fieldbook” Bantam Doubleday Dell Publishing Co.,

New York, New York, 1990 & 1994.

U.S. Government Satellite andLaunch Vehicle Failures from 1990 to ...2006.

www.aero.org/publications/crosslink/fall2007/04_side1.html

Weisinger, Hendrie, “Emotional Intelligence at Work: The Untapped Edge for Success”, 

Jossey-Bass Publishers, San Francisco, CA, 1998.
ix

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining 

the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing 

this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operation and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid 

OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To)

4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION

 REPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITOR’S ACRONYM(S)

11. SPONSORING/MONITORING REPORT NUMBER

12. DISTRIBUTION/AVAILABILITY STATEMENT

13. SUPPLEMENTARY NOTES

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF:

a. REPORT b. ABSTRACT c. THIS PAGE

17. LIMITATION OF ABSTRACT 18. NUMBER OF

 PAGES

19a. NAME OF RESPONSIBLE PERSON

19b. TELEPHONE NUMBER (Include area code)

Standard Form 298 (Rev. 8-98)

Prescribed by ANSI Std. Z39-18

Elements of Engineering Excellence

J.C. Blair, R.S. Ryan, and L.A. Schutzenhofer

George C. Marshall Space Flight Center, Huntsville, AL 35812

AI Signal Research, Inc., 3411 Triana Blvd. SW, Huntsville, AL 35805

National Aeronautics and Space Administration

Washington, DC 20546–0001

Unclassified-Unlimited

Subject Category 15

Availability: NASA CASI (443–757–5802)

Prepared for the Organization & Leadership Development Office, Office of Human Capital

COTR: James G. Andrews

M–1347

NNM07AA77C

Contractor Report May–December 2011

NASA/CR—2012–217471

launch/spacecraft vehicles, propulsion, design, testing, failure analysis

01–11–2012

U U U UU 60

The inspiration for this Contract Report (CR) originated in discussions with the director of Marshall Space Flight Center 

(MSFC) Engineering who asked that we investigate the question: “How do you achieve excellence in aerospace engineering?” 

Engineering a space system is a complex activity. Avoiding its inherent potential pitfalls and achieving a successful product is a 

challenge. This CR presents one approach to answering the question of how to achieve Engineering Excellence. We first investigated the ‘root causes’ of NASA major failures as a basis for developing a proposed answer to the question of Excellence. The 

following discussions integrate a triad of Technical Understanding and Execution, Partnership with the Project, and Individual 

and Organizational Culture. The thesis is that you must focus on the whole process and its underlying culture, not just on the 

technical aspects. In addition to the engineering process, emphasis is given to the need and characteristics of a Learning Organization as a mechanism for changing the culture.

STI Help Desk at email: help@sti.nasa.gov

STI Help Desk at: 443–757–5802
The NASA STI Program...in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA Scientific and Technical Information (STI) 

Program Office plays a key part in helping NASA 

maintain this important role.

The NASA STI Program Office is operated by 

Langley Research Center, the lead center for 

NASA’s scientific and technical information. The 

NASA STI Program Office provides access to 

the NASA STI Database, the largest collection of 

aeronautical and space science STI in the world. 

The Program Office is also NASA’s institutional 

mechanism for disseminating the results of its 

research and development activities. These results 

are published by NASA in the NASA STI Report 

Series, which includes the following report types:

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant 

phase of research that present the results of 

NASA programs and include extensive data 

or theoretical analysis. Includes compilations 

of significant scientific and technical data 

and information deemed to be of continuing 

reference value. NASA’s counterpart of peerreviewed formal professional papers but has less 

stringent limitations on manuscript length and 

extent of graphic presentations.

• TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis.

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees.

• CONFERENCE PUBLICATION. Collected 

papers from scientific and technical conferences, 

symposia, seminars, or other meetings sponsored 

or cosponsored by NASA.

• SPECIAL PUBLICATION. Scientific, technical, 

or historical information from NASA programs, 

projects, and mission, often concerned with 

subjects having substantial public interest.

• TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission.

Specialized services that complement the STI 

Program Office’s diverse offerings include creating 

custom thesauri, building customized databases, 

organizing and publishing research results...even 

providing videos.

For more information about the NASA STI Program 

Office, see the following:

• Access the NASA STI program home page at 

<http://www.sti.nasa.gov>

• E-mail your question via the Internet to 

<help@sti.nasa.gov>

• Fax your question to the NASA STI Help Desk 

at 443 –757–5803

• Phone the NASA STI Help Desk at 

443 –757–5802

• Write to:

NASA STI Help Desk

NASA Center for AeroSpace Information

7115 Standard Drive

Hanover, MD 21076–1320
National Aeronautics and

Space Administration

IS20

George C. Marshall Space Flight Center

Huntsville, Alabama 35812