# Presentation: 712906

## Managers who don’t know how to measure what they want
 settle for wanting what they can measure

Dr. Russ Ackoff at
 http://www.f-laws.com/pdf/A_Little_Book_of_F-LawsE.pdf

** ****settle for wanting what they can measure**
**Dr. Russ Ackoff at**
** **_[http://www.f-laws.com/pdf/A_Little_Book_of_F-LawsE.pdf](http://www.f-laws.com/pdf/A_Little_Book_of_F-LawsE.pdf)_

**Choosing Performance Indicators**

## Performance Indicator Introduction 

- It is more important how the measure is used than what the measure is
- Self-fulfilling prophecies can prevent us from gathering any data
- We are drowning in data, but little knowledge is derived
- Context and Operational Definitions are crucial

- SS Prevette

## Creating a Management System

- Dr. Ackoff, _Creating the Corporate Future_
- Three Management Functions:
- ID of actual and potential problems (threats and opportunities)
- Decision making (what to do and doing it, or having it done)
- Maintenance and improvement of performance under changing and unchanging conditions

- SS Prevette

## Five Critical Issues

- Managers suffer from overabundance of irrelevant information.
- Managers don’t know what information they need.  Need to look at the decision process to determine this.
- Even if given the information they need, decision making will not necessarily improve

- SS Prevette

## Five Critical Issues (4,5)

- More communication does not necessarily lead to better performance.  Information can be used destructively.
- Managers do need to know how the information system works.  Just because it came from a computer doesn’t mean it is right.

- SS Prevette

## Designing a Management System

- The information system should be designed as an integral part of the management system
- Most information systems are designed independently, leading to failure
- Information systems should serve management, not vice versa

- SS Prevette

## Dr. Ackoff’s message

- We do need to know the context within which the performance indicators will be used
- Forecasting and living with the forecasted future is important, but what about designing a better future?

- SS Prevette

## Barriers to PI Development - FEAR

- Higher ups will use it as a “hammer”
- Subjected to quotas and targets imposed from above
- Fear (“accountability”) used as a “motivator”
- Actions and Explanations as a result of random fluctuations
- Perceived loss of control over portrayal of performance
- Must develop “perfect” indicator the first time
- 			Use of SPC can minimize these fears

- SS Prevette

## Three Information Sources

- Worker and Customer Opinion
- Expert Review
- Process Measures
- We will focus on Process Measures for this topic.  Note that opinions can be converted to measurement data with survey analysis, and results can be converted to measurement data through grading criteria.

- SS Prevette

## Survey Processing Sidebar

- Surveys usually are not analyzed well
- Most arithmetically average 1 to 5 Likert scale and don’t asses for variation
- This assumes people think linearly
- This assumes each category is equal width
- Can over-react to random variation
- For a better idea, see _[http://](http://www.hanford.gov/safety/vpp/survey.htm)__[www.hanford.gov/safety/vpp/survey.htm](http://www.hanford.gov/safety/vpp/survey.htm)_

- SS Prevette

## Some Approaches for Choosing PI’s

- Top-Down
- Process Approach
- Bottom-Up
- Customer Focus
- Idealized Design
- Leading Indicators

- SS Prevette

## Top-Down Approach

- Look at your Mission and Vision
- What are your Products, Services and Customers
- What is your Business Objective
- What are desired Outcomes
- What are the Processes that accomplish the above (drawing a flow chart may help)
- Decide on Measures (see next page)
- Go set up data sources, gather data

- SS Prevette

## Process Approach

- REWORK

*Credit to Phil Monroe, DEMCOM*

- SS Prevette

## Connection from Output to Outcome

- Outcomes are only achieved as a result of a process
- Focusing ONLY on outcomes is a sure path to failure
- Ignoring outcomes is a sure path to failure
- When I provide a Product or a Service, what is my THEORY that connects this to a favorable outcome?
- Example – I provide statistical training to you.  My product is you, as you leave this room.  My theory is that you will apply the knowledge you have been provided, apply it to performance indicator work, which will cause continual improvement to occur and have a positive impact on accomplishing the Mission of your Organization.

- SS Prevette

## Examples

- Input Rate (units per time or Dollars, Hours)
- Efficiency (Input vs. Budget, Input vs. Idle)
- Cycle Time (Baldrige Criteria pushes cycle time)
- Backlog (inventory)
- Procedure Compliance, Completion without Stoppage
- Output Rate (units per time of Product or Service)
- Productivity (Output divided by Input)
- Defect Rate (Waste + Rework vs. Output)
- Effectiveness (Outcome measures, Outcome per Input, Percent Compliant, Output vs. Schedule)

- SS Prevette

## Bottom Up Approach

- Go find out what data you currently have
- Why are you collecting it?
- What could it tell you?
- Choose measures from available data
- Refine by trial and error
**Advantage – Cost Effective, utilizes existing resources**
**Disadvantage – Only focuses on “visible” data, is Reactive, Not “designed” (re: Ackoff)**

- SS Prevette

## Combination Approach

**1st**

**Work down from the Aim of the Organization**

**2nd**

**Work up from the data on hand**

**3****rd**** Meet in the middle, identify gaps**

- SS Prevette

## Customer Focus

- Put yourself in your customer’s place.
- What is important to the customer?
- Customer Satisfaction, Loyalty
- Can it be measured or inferred?
- Do the same for your employees.

- SS Prevette

## Idealized Design

- Your current organization and systems are gone
- What would you put in place TODAY?
- You are free to replace it with anything you like as long as:
- It is technologically feasible
- It must be operationally viable
- 						Dr. Russ Ackoff

- SS Prevette

## Low Frequency - High Impact Measures

- There are several “outcome” based measures such as Deaths, Bankruptcy, Property Damage, Environmental Damage which are infrequent occurrences but carry a high perceived risk and emotional reaction
- When one of these events occurs, we are susceptible to over reaction to the event itself

- SS Prevette

## Jump Start with Leading Indicators

- Just what are leading indicators, anyway?
- Predictions of future?
- or
- A means to create a better future?

- SS Prevette

## Creating a Better Future

- Distracted by calls to predict future, we delay development of leading indicators
- At low injury rate, little information exists in outcome indicators
- Trending response time is long at low rates
- Use leading indicators to measure lower threshold data and activities
- Quickens trend response and improves outcomes

- SS Prevette

## Leading and Lagging Indicators

- Lagging Indicators dominate at the higher levels, reflecting outcomes.  Tend to be standardized and dictated from above.
- Leading Indicators dominate at the lower levels, reflecting processes that achieve the outcomes.  Tend to be customized, and driven from the bottom-up.

- SS Prevette

## Hierarchy of Indicators

**Hierarchy of Indicators**

- Lagging indicators dominate at high levels, leading at lower levels.

- SS Prevette

## Performance Indicator Evolution

**As a process matures, one may end up evolving the indicators used.  For example, if interested in completing actions by commitment dates, one may end up using (as the process matures):**
- Percent of Actions completed by due date in effect at time of completion
- Percent of Actions completed without missing any due dates during their life
- Percent of Actions completed by the original due date
- Average days Actions completed ahead of original due date

- SS Prevette

## The Search for the “Perfect” Indicator

- When committees get together and try to table-top the perfect indicator, paralysis often sets in.
- Realize all data are flawed, there is no “true value”, indicators can always be “gamed.”
- Putting the right culture of HOW to use performance indicators in place minimizes adverse impacts.
- Gain experience with simple indicators, then move on to more complex indicators if needed.
- With proper analysis, flaws with existing data can be detected and fixed.  If you never look at the data, there will never be an incentive to fix the data.

- SS Prevette

## Three Kinds of Numbers for Management.

- Facts of life.  If we don't make this profit figure, we will go out of business.
- Planning, prediction and budget.  Can be used to compare alternative plans.
- Arbitrary numerical targets.  Generally used to judge workers by.
- Avoid the use of the 3rd kind of number
* **Henry Neave *_*The Deming Dimension*_

- SS Prevette

## Just Do It

- All data are flawed
- Make good use of your data
- Endless conference table discussions won’t cause any data to appear
- Initial prototype successes will lead to experience, and will further the spread of the use of indicators

- SS Prevette

## Plan ahead
Establish Operational Definitions
Check data quality
Use existing databases before creating new

- Establish Operational Definitions
- Check data quality
- Use existing databases before creating new

**Data Gathering**

- SS Prevette

## Plan Ahead

***“******It’s absolutely vital for business that you settle this method of counting, measuring, definition of faults, mistake, defect, before you do business.  It’s too late afterwards”***
**-Dr. W. Edwards Deming**
**How many initiatives have we embarked upon, without a clear set of indicators established up front, only to be left with, a year afterwards, trying to figure out “what happened”?**

- SS Prevette

## Operational Definitions

- “Clean the Table”

- SS Prevette

## Operational Definitions

- Most arguments about conflicting data come down to the definition of how to count the data
- Try to be precise in your definitions, but likely something unforeseen will arise
- Record the detail someplace:
- EM-RL—PHMC prefix occurrence reports
- Counted by categorization date
- Does not include canceled reports

- SS Prevette

## Context

- Do not look at a chart in a vacuum
- Reconcile any differences between the data and “gut feeling”
- Combine experience and the data
- Lessons from the data should lead to insight in the field, and vice versa

- SS Prevette

## Data Quality

- Data should be replicable
- Operational Definitions are a must
- Source Data must be defined
- There is no “true value” of any measure, but a good operational definition can save much trouble in the future
**ANYONE at ANYTIME in the future should be able to apply the same operational definition to the same source data, and get the same results.**

- SS Prevette

## Conclusion:

- Performance Indicators are part of the Management System
- Good use of data will encourage Performance Indicator Development
- All data are flawed – Set up good Operational Definitions to minimize flaws
-  “Just Do It” – Start collecting data and use it
- There is no such thing as a bad performance indicator, only bad use of performance indicators
- Good use of performance indicators will lead to continual improvement

- SS Prevette