# Presentation: 168196

## The Art of the Trade Study

**Mark A. Wilson**
- Strategy Bridge International, Inc.
- 12020 Sunrise Valley Dr, Suite 100
- Reston, VA 20191-3429
- (703) 476-2252
- FAX: (703) 832-0292

## My Purpose Today

- Explain the value of a defined methodology for conducting engineering trades
- Describe a proven trade study process
- Explain how the Analytical Hierarchy Process (AHP) methodology can be applied to your trade studies

## How are your decisions typically made?

- BOGGSAT?

**Notes:**
Procedural decision aids
Analytic decision aids
Rules of Thumb (heuristics)

Bunch Of Guys and Gals Sitting Around Talking

## Context

- Systems Engineers constantly define, prioritize, and decide programmatic, technical and life cycle concerns
- Proper technical decision making must balance:
- Performance
- Cost effectiveness
- Schedule
- Supportability
- The Trade Study is a _core SE skill_
- Provides a repeatable, efficient method for visible, traceable, justifiable, decisions

**Notes:**
Decision Analysis activities provide the basis for evaluating and selecting alternatives when decisions need to be made.  Decision Analysis involves selecting the criteria for the decision and the methods to be used in conducting the analysis.   For example, during system design, analysis must be conducted to help chose amongst alternatives to achieve a balanced, supportable, robust, and cost effective system design.  These analysis include, but are not limited to, trade studies, models and simulation, supportability analysis, level of repair analysis, post fielding support analysis, repair vs discard, and cost analysis.  These studies should be augmented with virtual and/or physical prototypes, where applicable, prior to making decisions on best alternative.  Decision criteria will be influenced by such things as interoperability constraints; size; transportability requirements; maintenance concept; affordability; reliability, availability, and maintainability goals; and schedule.

Chapter 4 DoD 5000.2 systems engineering guide

## When trade studies go bad...

**When trade studies go bad...**

## Real-World Trades

- How do _you _deal with:
- Complexity
- Conflicting objectives and multiple alternatives
- Overload
- Trying to consider numerous factors at once
- Implicit assumptions
- “Seat of the pants” conclusions
- Engineering Team Buy-in
- Decisions with lukewarm support

## Slide 7

**Notes:**
Failure to recognize a problem in time
Procrastination
Failure to recognize a paradigm shift
Preoccupation with fire-fighting
Internal politics
Lack of will
Lack of urgency
Organizational myopia
Inadequate data
Too much data
Failure to delegate
Incompetence
Indecisiveness
Too many people involved in the decision
Decision bottlenecks
Personal bias or prejudice
Inaccurate or incomplete information
Arrogance
Complacency
Fear of mistakes
Decision delayed until too late
Problem ignored or hidden
Lack of data
Number of people – too many
Hidden agenda
Lack of authority
Resistance to change

## How do engineers really make decisions?

**Analytic Processing**
**(Rational Analysis)**

**Automatic Processing**
**(Intuition)**

***How often do you use your intuition to make engineering decisions?***

## The Trouble with Intuition

**Common Cognitive Biases in Engineering Decisions**
- Group Think
- Status Quo Bias
- Overconfidence Bias
- Wishful Thinking Errors
- Input Bias
- Confirming Evidence Bias

_***Common Effects***_***: ***
* * Failure to critically examine all alternatives
-  Tendency to continue to “do things the way we we’ve done them.”
-  Illusion of control over stochastic events
-  Overestimation of probability of desired outcomes

**Notes:**
Independent events have no memory.

## Intuition Test

- Aoccdrnig to rscheearch at an Elingsh uinvervtisy, it deosn’t mttaer in what oredr the ltteers in a wrod are, olny that the frist and lsat ltteres are at the rghit pcleas.  The rset can be a toatl mses and you can sitll raed it wouthit a porbelm. This is bcuseae we do not raed ervey lteter by ilstef, but the word as a wlohe.

**Notes:**
Strength – able to make sense of jumbled things.
Weakness – miss things because do not see the details.

## How does “stress” affect decision-making?

_**Principle of “Constancy”**_
- Disruption of stable relations with environment is perceived as a “threat” that induces a cognitive response to reestablish stability
-  Loss of constancy = stress, which triggers SNS “fight or flight”

_**Effects of Stress **_
-  Occupies working memory
-  Emotion tends to dominate reason
-  Disrupts cognitive processes, especially  complex information processing
-  Encourages heuristic thinking
-  Attention is focused on one or two salient cues
-  Disproportionate weight ascribed to negative information
-  Tendency to lock in and defend the first chosen strategy

**Notes:**
SNS response: digestion shuts down, arteries dilate and blood vessels direct blood to body core and away from extremities
PNS “backlash” explains why people are very sleepy after very stressful events

## The Case for a Consistent Trade Study Methodology

**Provides a better *****expected***** outcome than random choice**
- We can’t control outcomes; the best we can do is influence the probability of certain outcomes
**Overcome cognitive biases**
- Mitigate negative effects of intuition
**Permits decision traceability**
- Allows decision process improvement
**Builds justification and helps others understand reasoning **
- More likely to influence up the management chain

***   ******As I think back over the years, I have been guided by four principles in decision making. First, the only certainty is that there is no certainty. Second, every decision, as a consequence, is a matter of weighing probabilities. Third, despite uncertainty we must decide and we must act. And lastly, we need to judge decisions not only the results but on how they were made.***
*** ***                                                                                                                         Robert Rubin, 1999

**Notes:**
Film – Hunt for Red October - 1:50 – decision about dealing with the shot torpedo.

## Purpose of a Structured Trade Study

- Reach the _right decision_ for the _right reasons_
- Don’t settle for just *any* decision because your team is too exhausted to argue any longer

***  ******Making tradeoffs is a fact of organizational life, especially in a resource-constrained environment. ...priorities must be determined on the basis of the enterprise's overall objectives.   ***

## Trade Study Flow Chart

- Identify the decision situation-
- understand the objectives
- “Framing”

- Identify
- Alternatives

- Model-Evaluate
-  the Decision

- Choose an
- Alternative

- Conduct
- Sensitivity
- Analysis

- Analyze
- Further?

- Implement the Selected
- alternative

- No

- Yes

## “Figuring out how to think

-   When asked what single event was most helpful
- in developing the theory of relativity,
- Albert Einstein reportedly answered:*       *

*“**Figuring out how to think*
*about the problem”*

- <date/time>

**Notes:**
Framing is about creativity.

## The Master Decision-Making Skill: Framing

*Context* is the set of alternatives appropriate to a specific situation

**How you **_**define**_** a problem will largely determine how you will go about **_**solving**_** it**

**Frame = Decision Context**

**Notes:**
Framing is how you define the problem.  It’s the first step in the process, therefore it has tremendous leverage on all the other steps.

How you frame the problem defines the set of alternatives you will consider.  This is a critically important point.

Sounds easy, doesn’t it?  But we’re going to show you that it can be trickier than think.  Later on I’m going to share some examples of how very intelligent people solved the wrong problem, which wound up costing lots of money, and lost lives as well.

## Understanding Your Frame

**Overall Fundamental Objective**

**X**

**(Specific Solutions)**

**Fundamental Objectives**
**“****Decision Criteria”**

**Notes:**
What’s another example.  You are struggling to find COTS components which meet your requirements.  Rather than focus on looking all over the world for yet another supplier, maybe the fundamental question of whether to design some of the parts in-house should be reconsidered.

## Helpful Questions

- What is your ultimate objective?
- What is the crux of the issue or problem?
- How does the decision affect other decisions?
- What information do we have about similar problems or decisions made previously?
- How will we implement the solution?

***  “******There are no decision aids that can structure a problem automatically.  Rather, this crucial phase must be largely achieved through unaided human judgment”  ***
***                                           ******- Judgment and Choice, Robert Hogarth***

**Notes:**
Ultimate objective:  highest level

Ask someone with a fresh perspective for their opinion

## Abstract ideas that influence thinking and action in the organization
Behaviors that are celebrated or rewarded reflect true values 

- Behaviors that are celebrated or rewarded reflect true values

- Culture and Risk Philosophy are _crucial_ to the decision frame

** ****Context of Implementation will Drive:**
** **Design
-  Design Margin
-  Reliability
-  Quality

**Notes:**
IBM 

## How does your project context affect your decision frame?

**Missed opportunity results**
**in worse consequences**

**Inappropriate choice is more damaging**

- Neutral Bias

***You must assess the costs of potential negative outcomes for each type of error***

**Notes:**
MINIMAX vs MAXIMIN

## Technique: Means Objective Network

- Station Keeping
- for Entire
- Constellation

- Decrease Risk
- Of Losing
- New Assets

- Option
- Confidence

- System can
- Expand with
- Constellation

- Supportable
- Solution

- Operators
- Understand
- System

- Produce
- Adequate
- Documentation

- Integrate into
- Existing
- Systems

- Maintain
- H/W

- Maintain
- S/W

- System to come
- On line quickly

**Notes:**
Brainstorm all objectives then ask “why is that important” to tie all objectives back to the “fundamental” reason.  Tie together to find the originating concept – because it is!

Look at page 44 in text book.

## Trade Study Flow Chart

- Identify the decision situation-
- understand the objectives
- “Framing”

- Identify
- Alternatives

- Model-Evaluate
-  the Decision

- Choose an
- Alternative

- Conduct
- Sensitivity
- Analysis

- Analyze
- Further?

- Implement the Selected
- alternative

- No

- Yes

## Generating Alternatives

**“****Alternatives are the raw material of decision making”**** **
**   ****-*****Smart Choices *****by Hammond, Keeney & Raiffa**

**After the problem has been framed, ask:**
**“****How can we obtain the desired outcome?”**

- Challenge constraints – look at the problem from new angles
- Be creative, let process diverge
- Gather information, if necessary
- Withhold judgment until the evaluation phase

## How do you ensure that you are not considering the same old alternatives while falling into a “status quo” trap?

**How do you ensure that you are not considering the same old alternatives while falling into a “status quo” trap?**

**Notes:**
Failure to recognize a problem in time
Procrastination
Failure to recognize a paradigm shift
Preoccupation with fire-fighting
Internal politics
Lack of will
Lack of urgency
Organizational myopia
Inadequate data
Too much data
Failure to delegate
Incompetence
Indecisiveness
Too many people involved in the decision
Decision bottlenecks
Personal bias or prejudice
Inaccurate or incomplete information
Arrogance
Complacency
Fear of mistakes
Decision delayed until too late
Problem ignored or hidden
Lack of data
Number of people – too many
Hidden agenda
Lack of authority
Resistance to change

## Traps to Avoid

- Considering only a single alternative
- Considering only conventional, or “business as usual” approaches
- Looking for the “perfect” solution

## Trade Study Flow Chart

- Identify the decision situation-
- understand the objectives
- “Framing”

- Identify
- Alternatives

- Model-Evaluate
-  the Decision

- Choose an
- Alternative

- Conduct
- Sensitivity
- Analysis

- Analyze
- Further?

- Implement the Selected
- alternative

- No

- Yes

## Most Decision Problems are Multicriteria

-  Satisfy science requirements
-  Maximize design life
-  Minimize lifecycle cost
-  Maximize reliability
-  Minimize costs of production
-  Satisfy political stakeholders

*Decision Criteria: *
- The means by which a decision-maker measures the attributes of alternatives in order to identify and assess discriminators

## How do you compare objective and subjective measures?

- How do you compare things with intangible properties?

-  Taste
-  Aroma
-  Acidity
-  Price
-  ?

***Can you compare apples to oranges?***

_***Question***_***: ***
- Do political considerations ever factor in your trade studies?

**Notes:**
Taste  is not an objective measure

Apple vs. orange - get someone to choose one of them.

## AHP Methodology in Trade Studies

**Prioritizes multiple tangible and intangible criteria:**
**  ****In most decisions, intangibles such as**

** **political factors and
-  social factors

**take precedence over tangibles such as**

** **economic factors and
-  technical factors

-   It is not the precision of measurement on a particular factor
-      that determines the validity of a decision, but the _importance_
-      we attach to the factors involved
-   AHP assigns importance to all the factors and _synthesizes_
-      this diverse information to make the best decision

***Example: ****the decision to use aluminum instead of a titanium alloy for the Boeing 777 wings was not the technically preferred alternative.*

**Notes:**
AHP is useful when the complexity of the synthesis required exceeds our cognitive abilities

Allows ignoring the absolute numbers and comparing the “feelings” about the relative importance of the various parameters.

Real decisions are not objective measures.

**CRITERIA**

- The Analytic Hierarchy Process
- (Saaty- 1971)

**ALTERNATIVES**

**Notes:**
Power of AHP: assigning weights based on pair-wise comparisons to create ratios. Or can do ratings model.
Ratings model – relative weights of criteria then evaluate how well an alternative satisfies a criteria.  Set a rating scale (make sure it is defined). 

## Selecting Criteria

- High quality evaluation criteria:
- Are linked to the critical aspects of the solution
- “Value” areas
- Risk areas
- Are limited to those that will yield meaningful discrimination between solutions
- Are reasonably independent of each other

## Remember Our Means Objective Network?

- Station Keeping
- for Entire
- Constellation

- Decrease Risk
- Of Losing
- New Assets

- Option
- Confidence

- System can
- Expand with
- Constellation

- Supportable
- Solution

- Operators
- Understand
- System

- Produce
- Adequate
- Documentation

- Integrate into
- Existing
- Systems

- Maintain
- H/W

- Maintain
- S/W

- System to come
- On line quickly

## Establish a Fundamental Objectives Hierarchy

**Operability**

**Decision Frame**

**Maintainability**

**Reliability**

**Schedule**
**Risk**

**Growth **
**Potential**

** ****Select a New**
**Telemetry Solution**

**Option **
**Confidence**

**Notes:**
May not want to include prices as a criteria since it will be considered in the final decision making.  

## Choose a New Telemetry Solution for our satellite constellation

**Operability**
- measure of the ease with which an operator can comprehend the operating concept and operate the equipment
**Maintainability**
- Mean Time to Repair (MTTR)
**Reliability**
- Mean Time Between Failures (MTBF)
**Option Confidence (i.e., will work as advertised)**
- degree to which equipment will operate properly with spacecraft
- product maturity
**Schedule Risk**
- risk that option can be delivered in time to meet next launch requirement
**Growth Potential**
- to be able to have a device which is modular and readily expandable and upgrade-able
- capability of the design to readily accommodate technology insertion

## AHP Approach

- How does AHP capture human judgments?
- AHP does _not_ require you to make an absolute judgment or assessment.
- Process uses relative assessment between _two items at a time_.
- In relative measurement a preference, judgment is expressed on each pair of elements with respect to the common “parent” element.

## Simplified AHP Criteria Weighting Matrix

- Technique
- Compare a list of items to one another to determine relative importance
- Uses
- Developing criteria weights
- Reveals high impact factors

**Factor**
** ****A**

**Factor**
** ****B**

**Notes:**
Analytic Hierarchy Process
High Impact means they drive the decision. E.g., a factor with a 2% weight is inconsequential compared to one with 40%

Fundamental benefit of pair wise – it is easier (1-1) and harnesses intuition.

## Example Top Level Criteria Weights

- 	**1****	****Equal importance**
**	****3****	****Moderate importance of one over another**
**	****5****	****Strong or essential importance**
**	****7****	****Very strong or demonstrated importance**
**	****9****	****Extreme importance**
**	****2,4,6,8****	****Intermediate values**

**Notes:**
Dr. Tom Saaty, originator, software tool “expert choice”

Compare rows to columns

1 = Equal
5 = strong preference
9 = extreme preference

Do half the matrix, then fill in the inverse on the other side.

Across the median blanks are the inverse.

Weight column:  row total divided by grand total.  This gives you the relative weighting.

Might throw out safety because it is under 5% - makes one less calculation and will not affect the outcome.

## GFE Option

**Supplier A**

**Supplier B**

**Operability**
**13%**

** ****GOAL**

**Maintainability**
**10%**

**Reliability**
**20%**

**Schedule**
**Risk**
**3%**

**Growth **
**Potential**
**15%**

** ****Select Telemetry Solution**

**Option **
**Confidence**
**39%**

## Build Rating Scales

- Qualitative Scale (with weights)
- Excellent = 100%
- Acceptable = 60%
- Marginal = 30%
- Not Addressed = 0%

## Rating Alternatives

## Summary of AHP Advantages

- Easy to use in trade studies
- Organizes, prioritizes and synthesizes complexity within a rational framework
- Breaks down tangible and intangible criteria into manageable components
- Fosters critical discussion and examination of  implicit assumptions when used with diverse groups
- Makes it possible to deal with conflicts in perception and in judgment

## Trade Study Flow Chart

- Identify the decision situation-
- understand the objectives
- “Framing”

- Identify
- Alternatives

- Model-Evaluate
-  the Decision

- Choose an
- Alternative

- Conduct
- Sensitivity
- Analysis

- Analyze
- Further?

- Implement the Selected
- alternative

- No

- Yes

## Sensitivity analysis answers the question: “What makes a difference in this decision?”

Sensitivity analysis determines whether small changes in judgments affects the final weights and rankings of the alternatives. 
May lead the decision-maker to reconsider the decision frame or the sufficiency of alternatives 

If so, the decision-maker may want to review the sensitive judgments.

- Sensitivity analysis determines whether small changes in judgments affects the final weights and rankings of the alternatives.
- May lead the decision-maker to reconsider the decision frame or the sufficiency of alternatives
- If so, the decision-maker may want to review the sensitive judgments.

- Sensitivity Analysis

## Technique: Two-Way Sensitivity Graph (Comparison of One Factor to Another)

**Worst**

**Most **
**Expensive**

**Cheapest**

**Best**

**Technical Merit**

**Cost**

**Sensitivity Line**

**1**

**0**

**0**

**1**

**50%**

**Better**

**Better**

**Notes:**
Do it this way if comparing contradictory views – technical performance versus price for instance.  

## Other Applications for This Methodology

- Portfolio Selection and Resource Optimization

## Why have a process for Project Portfolio Selection – Resource Optimization?

- Prioritize initiatives in a systematic way
- Optimize *overall* organizational benefit
- Meet all funding constraints
- Create a prioritized list of unfunded requirements should more budget become available

## Strategy Bridge Recommended Process

- Convene the decision team with a skilled facilitator
- Agree on the fundamental objective, key business drivers, and supporting decision criteria
- Create a shared understanding of the alternatives under consideration
- Build a decision model
- Decision Lens COTS software solution for group decision-making based on the Analytic Hierarchy Process
- Provides a method for quickly synthesizing qualitative and quantitative information from multiple stakeholders for strategic trade-offs
- Facilitate a collaborative decision process and sensitivity analysis
- Document assumptions, criteria, and decision results

**Notes:**
Use facilitator – objective, not concerned about the outcome, no content decisions – manages the process of the meeting.
Facilitator knowledge is a benefit (domain knowledge) but not absolutely necessary.

## Benefits

- Maximize ROI
- Align project portfolio with organizational objectives
- Increase Decision Visibility
- Document and examine implicit assumptions
- Track, audit and improve decision-making over time.
- Create Decision Traceability
- Final decision authorities can see how decisions were achieved
- Adjust strategy to changing market conditions with dynamic sensitivity analysis
- Save Time
- Active participation of all decision-makers minimizes the need for rework or to “sell” the decision later
- Reduce management frustration and eliminate endless debate

## Example – What do we fund this year?

- Some criteria to consider
- Provide new services
- Enhance customer satisfaction
- Cost avoidance
- Maximum ROI
- Etc.

## Summary

**Why have a consistent trade study methodology?**
- Overcome negative aspects inherent with intuition (cognitive biases)
- Permit decision traceability
- Build justification and help others understand reasoning
- Improve your trade study process
**Why use AHP?**
- Does not require absolute assessments
- Permits comparisons of tangible and intangible factors
- Fosters rich dialog among engineering team members
- Straight-forward and easy to use

## Questions?

**Systems Engineer**** **
**“****Toolkit”**

**Configuration Management**

**Test Engineering**

**Risk Management**

**Cost Analysis**

**Team Leadership**

**Technical Decision-Making**

**Integration**

**Modeling & Simulation**

**Technical Planning**

**Requirements Management**

**Architecting**

**Technology Assessment**

**CONOPS Formulation**