# Presentation: 923485

## The Secretary’s Robert W. CareyPerformance Excellence Awards  

- New Examiner & Refresher Training
- Apr 29, 30 & May 1, 2008

- Version 5.0    April 2008

- File 1 of 3

## General Information

- Emergency telephone number
- Messages
- Rest rooms
- Food
- Start/stop times
- Questions???

- Version 5.0    April 2008

- File 1 of 3

## Ground Rules

- Respect opinions of others
- All questions are good questions
- What is said in class – stays in class
- Back from breaks timely
- Work hard and have FUN
- Others?

- Version 5.0    April 2008

- File 1 of 3

## Carey Office and Instructors

- Carey Program Office
- Scott Holliday, Director, Management Systems Improvement Service, VACO, 008B3
- Eric Malloy, Program Manager, Performance Excellence
- Gwendolyn Young, Patrick Hyousse, & Tom Garin
- Instructors
- Long Beach, CA: Kurt Gundacker
- Washington, DC: Leo Calderone & Kathleen Testa

- Version 5.0    April 2008

- File 1 of 3

## Michael Greenblatt, Program Analyst, VHA
Lindee Lenox, Director, Memorial Cemetery Service, NCA
Jeffrey London, Management & Program Analyst, VBA 

- Michael Greenblatt, Program Analyst, VHA
- Lindee Lenox, Director, Memorial Cemetery Service, NCA
- Jeffrey London, Management & Program Analyst, VBA

- Version 5.0    April 2008

- File 1 of 3

## Pre-Test

- Purpose – To determine subject matter knowledge before in-class training begins
- Individual test scores will not be given to your supervisor
- 20 questions, 5 points each
- No penalty for guessing
- 15 minutes - maximum

- Version 5.0    April 2008

- File 1 of 3

## Introductions

- Pick a partner and gather the following information (Approximately 5 minutes)
- Name
- Organization
- Baldrige or Carey criteria experience
- Something others might not know about them
- Introduce your partner to the class (Approximately 30 seconds to one minute)

- Version 5.0    April 2008

- File 1 of 3

## Administrative Items, Pre-Test, and Introductions
Carey Examination Process
Conflict of Interest & Core Values
Baldrige Framework and Criteria Book
Organizational Profile and Key Factors
Process Scoring Aspects and Item Requirements
Category 3 and Exercises
Strength Comments
Summary and Feedback

- Administrative Items, Pre-Test, and Introductions
- Carey Examination Process
- Conflict of Interest & Core Values
- Baldrige Framework and Criteria Book
- Organizational Profile and Key Factors
- Process Scoring Aspects and Item Requirements
- Category 3 and Exercises
- Strength Comments
- Summary and Feedback

- Version 5.0    April 2008

- File 1 of 3

## Review
Category 3 and Exercises
Opportunity for Improvement Comments
Category 7 and Results Scoring Aspects
Item 7.2 and Exercises
Category 2 and Exercises
Item 7.1 and Exercises
Category 1 and Exercises
Summary and Feedback

- Review
- Category 3 and Exercises
- Opportunity for Improvement Comments
- Category 7 and Results Scoring Aspects
- Item 7.2 and Exercises
- Category 2 and Exercises
- Item 7.1 and Exercises
- Category 1 and Exercises
- Summary and Feedback

- Version 5.0    April 2008

- File 1 of 3

## Review
Item 7.6 and Exercises
Scoring an Application
Examination Week Simulation
Category 5 and Item 7.4
Themes
Completing a Scorebook
Site Visits
Post-Test & End of Course Survey

- Review
- Item 7.6 and Exercises
- Scoring an Application
- Examination Week Simulation
- Category 5 and Item 7.4
- Themes
- Completing a Scorebook
- Site Visits
- Post-Test & End of Course Survey

- Version 5.0    April 2008

- File 1 of 3

## Carey 

- End

- End

- (5a)
- Finalize,
- Send to
- applicant

- (8a)
- Finalize,
- Send to
- applicant

- (8) Array by score

- Y

- (3)
- Examiner
- team consensus

- (5)
- Site
- Visit?

- (9) Recommendations Sent to Secretary

- (6)
- Clarify/verify
- information

- (1)
- MSIS receives
- application

- (2)
- Examiner team
- independent review

- N

- Y

- Carey
- Examination
- Process
- Overview

- N

- (4)
- Judge review &
- Calibration

- (7) Other checks

- Version 5.0    April 2008

- File 1 of 3

## Examination - Scorebook Development Process

**Conflict of**
**Interest**

**Assemble**
**Materials**

- Draft Initial
- Key Themes

- Draft Initial
- Key Factors

- Identify KFs
- And Draft
- Comments

- Evaluate
- Criteria
- Item

- Update KFs

- Score the
- Item

- Repeat these
- Steps for each
- Criteria Item

- Complete
- Scorebook

- Finalize Key
- Themes

- Review Criteria
- And Read
- Application

- Version 5.0    April 2008

- File 1 of 3

## Conflict of Interest/ Confidentiality Statement 

- As a member of the Carey Board of Examiners, I affirm that I am not aware of any personal conflict of interest with this applicant. I will not disclose any information gained through the evaluation of the applicant about the applicant; the applicant’s clients, competitors, customers, or suppliers; or any other associated person or organization to anyone other than those in the Carey Program directly involved with the applicant review process.
- I understand that I am responsible for the information entrusted to me to conduct this examination.  At the completion of my participation in this program I will return the application and all related examination notes to the Carey Program office.

- Version 5.0    April 2008

- File 1 of 3

**Notes:**
Affirm there is no conflict of interest with the applicant

## Code of Ethical Standards/ Declaration of Principles 

- In promoting high standards of public service and ethical conduct, board members:
- shall conduct themselves professionally, with truth, accuracy, fairness, respect, and responsibility to the public
- shall not represent conflicting or competing interests nor place themselves in such a position where the board member’s interest may be in conflict, or appear to be in conflict, with the purposes and administration of the Award
- shall safeguard the confidences of all parties involved in the judging or examination of present or former applicants

- Version 5.0    April 2008

- File 1 of 3

**Notes:**
Do not disclose any information about the applicant gained through the evaluation process to anyone outside the applicant review process
Members of the Carey Board of Examiners pledge to uphold their professional principles in the fulfillment of their responsibilities as an examiner

## Code of Ethical Standards/ Declaration of Principles  

- In promoting high standards of public service and ethical conduct, board members (continued):
- shall not offer confidential information or disclosures that may in any way influence the Award integrity or process, currently or in the future
- shall not serve any private or special interest in fulfillment of the duties of a Judge or Examiner, therefore excluding, by definition, the examination of any organization or subunit of an organization by which he/she is employed or with which a consulting arrangement is in effect or anticipated
- shall not intentionally communicate false or misleading information that may compromise the integrity of the Award process or decisions therein

- Version 5.0    April 2008

- File 1 of 3

## Criteria for Performance Excellence
Application
Reference Material (such as books by Mark Graham Brown, Mark Blazey, John Vinyard) 
Materials for Taking Notes
A Quiet Place to Work
Time

- Criteria for Performance Excellence
- Application
- Reference Material (such as books by Mark Graham Brown, Mark Blazey, John Vinyard)
- Materials for Taking Notes
- A Quiet Place to Work
- Time

- Version 5.0    April 2008

- File 1 of 3

## Examination - Scorebook Development Process

- Conflict of
- Interest

- Assemble
- Materials

- Draft Initial
- Key Themes

- Draft Initial
- Key Factors

- Identify KFs
- And Draft
- Comments

- Evaluate
- Criteria
- Item

- Update KFs

- Score the
- Item

- Repeat these
- Steps for each
- Criteria Item

- Complete
- Scorebook

- Finalize Key
- Themes

**Review Criteria**
**And Read **
**Application**

- Version 5.0    April 2008

- File 1 of 3

## Baldrige Criteria for Performance Excellence

- What are the “Criteria”?
- Set of  questions that assess actions taken by successful businesses that drive high performance
- Fact-based evidence
- Applicable to all types of organizations
- Large & small
- Manufacturing
- Service
- Healthcare
- Education
- Government and Non-profit

- Version 5.0    April 2008

- File 1 of 3

## Organizational Profile:

- Environment, Relationships, and Challenges

- I
- Leadership

- 7
- Results

- 4
- Measurement, Analysis, and Knowledge Management

- 2
- Strategic
- Planning

- 3
- Customer and
- Market Focus

- 6
- Process
- Management

- 5
- Workforce Focus

**Baldrige Framework – A Systems Perspective**

- Version 5.0    April 2008

- File 1 of 3

## Visionary Leadership
Customer or Patient-Focused Excellence
Organizational and Personal Learning
Valuing Workforce Members and Partners
Agility
Focus on the Future
Managing for Innovation
Management by Fact
Social Responsibility & Community Health
Focus on Results and Creating Value
Systems Perspective

- Customer or Patient-Focused Excellence
- Organizational and Personal Learning
- Valuing Workforce Members and Partners
- Agility
- Focus on the Future
- Managing for Innovation
- Management by Fact
- Social Responsibility & Community Health
- Focus on Results and Creating Value
- Systems Perspective

- Criteria Are Built on a Set of Interrelated Core Values and Concepts

- Version 5.0    April 2008

- File 1 of 3

## The Role of Core Values and Concepts

- Version 5.0    April 2008

- File 1 of 3

## Read the Core Values on pages 48-53 (healthcare)
Each table or group will be given a different core value to read and report on to the class
Discuss the core values in your group:
Determine what the criteria describes as core value
Why is it difficult for organization to implement?
What would you look for on a site visit?
What is an example of this core value?

- Read the Core Values on pages 48-53 (healthcare)
- Each table or group will be given a different core value to read and report on to the class
- Discuss the core values in your group:
- Determine what the criteria describes as core value
- Why is it difficult for organization to implement?
- What would you look for on a site visit?
- What is an example of this core value?

- Version 5.0    April 2008

- File 1 of 3

## Baldrige Criteria for Performance Excellence

- Item listing with Point Values (p. 3)
- Preface: Organizational Profile (p. 4-6)
- Criteria for Performance Excellence (p. 7-26)
- Key Characteristics of the Criteria and changes from 2007 (p. 27)
- Criteria Response Guidelines (p. 29-32)
- Criteria: Category and Item Descriptions (p. 33-47)
- Criteria: Core Values & Concepts (p. 48-53)
- Glossary of Key Terms (p. 56-64)
- Scoring System & Scoring Guidelines (p. 65-68)

- Version 5.0    April 2008

- File 1 of 3

## Quickly read it from cover-to-cover
As you read it a second time, mark it up and put notes in the margins
Identify cross-references
Identify linkages:
Organizational profile to processes
Processes to results
Organizational profile to results

- Quickly read it from cover-to-cover
- As you read it a second time, mark it up and put notes in the margins
- Identify cross-references
- Identify linkages:
- Organizational profile to processes
- Processes to results
- Organizational profile to results

- Version 5.0    April 2008

- File 1 of 3

## Examination - Scorebook Development Process

- Conflict of
- Interest

- Assemble
- Materials

- Draft Initial
- Key Themes

- Review Criteria
- And Read
- Application

**Draft Initial**
**Key Factors**

- Identify KFs
- And Draft
- Comments

- Evaluate
- Criteria
- Item

- Update KFs

- Score the
- Item

- Repeat these
- Steps for each
- Criteria Item

- Complete
- Scorebook

- Finalize Key
- Themes

- Version 5.0    April 2008

- File 1 of 3

## Organizational Profile

- Snapshot of the applicant’s organization
- Sets Context for later Items
- Two Parts
- Organizational Description
- Organizational Challenges
- Starting point in understanding what’s important to the applicant organization
- Helps identify key gaps in information, requirements, or results
- Administratively used by Examiners or Judges:
- Understand the organization
- Understand the basis of comments (+/-)

- Version 5.0    April 2008

- File 1 of 3

## Baldrige Bookends

| What is the customer segment or group? (P1b(2)) | What are the customer requirements and expectations for products and services? (P1b(2)) | How are we performing against the customer requirements? (7.1) | Is the customer satisfied? (7.2a(1)) | Is the customer loyal? (7.2a(2)) |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
|  |  |  |  |  |

- Version 5.0    April 2008

- File 1 of 3

## Key Factors (KF) Worksheet

- _Definition_ - attributes of an organization and its environment that help assess performance
- Most important elements or factors, those that are critical to achieving intended outcomes
- Essential elements for pursuing or monitoring a desired outcome
- _Purpose_ - provide a concise summary of the most important aspects of the applicant’s environment

- Version 5.0    April 2008

- File 1 of 3

## Worksheet organized into the following five sections: 
Organizational Environment
Organizational Relationships
Competitive Environment
Strategic Context
Performance Improvement System

- Organizational Environment
- Organizational Relationships
- Competitive Environment
- Strategic Context
- Performance Improvement System

- Key Factors Worksheet

- Version 5.0    April 2008

- File 1 of 3

## Using Key Factors to Examine an Item

- Identify KFs that are associated with Criteria Item being worked
- Evaluate criteria responses using KFs as a tool for understanding how criteria relate to applicant’s organization
- KEY POINT – As an _examiner_, you want to look for these linkages to KFs and make sure responses address them
- Since criteria are generic, KFs relate criteria to applicant’s specific organization
- For example, if KFs identify three customer segments then “Customer Focus” responses should discuss those three segments. Customer Focused Results should also address the requirements of these same three customer segments. Links to other criteria responses may also be found.

- Version 5.0    April 2008

- File 1 of 3

## Read Organizational Profile from Criteria
Read Organizational Profile in Case Study
Identify teams and assign areas for review
Teams identify Key Factors for assigned area and put them on flip charts
Hang charts on wall

- Read Organizational Profile in Case Study
- Identify teams and assign areas for review
- Teams identify Key Factors for assigned area and put them on flip charts
- Hang charts on wall

- EXERCISE – Developing a Key Factors Work Sheet

- Version 5.0    April 2008

- File 1 of 3

## Examination - Scorebook Development Process

- Conflict of
- Interest

- Assemble
- Materials

**Draft Initial**
**Key Themes**

- Review Criteria
- And Read
- Application

- Draft Initial
- Key Factors

- Identify KFs
- And Draft
- Comments

- Evaluate
- Criteria
- Item

- Update KFs

- Score the
- Item

- Repeat these
- Steps for each
- Criteria Item

- Complete
- Scorebook

- Finalize Key
- Themes

- Version 5.0    April 2008

- File 1 of 3

## Key Themes

- A key theme is a high-level strength, opportunity for improvement, or result that:
- Is common (cross-cutting) to more than one item/category;
- Is significant in terms of the applicant’s Key Factors;
- Addresses a Core Value of the Criteria; and/or
- Is repeated multiple times in the application

- Version 5.0    April 2008

- File 1 of 3

## Core Values
Organizational Profile (or Key Factors)
Repetitive comments or a very important comment (strength or OFI)
Repetitive questions or gaps
Key capabilities or enablers
Key linkages

- Core Values
- Organizational Profile (or Key Factors)
- Repetitive comments or a very important comment (strength or OFI)
- Repetitive questions or gaps
- Key capabilities or enablers
- Key linkages

- Version 5.0    April 2008

- File 1 of 3

## Examination - Scorebook Development Process

- Conflict of
- Interest

- Assemble
- Materials

- Draft Initial
- Key Themes

- Review Criteria
- And Read
- Application

- Draft Initial
- Key Factors

**Identify KFs**
**And Draft**
**Comments**

**Evaluate**
**Criteria**
**Item**

- Update KFs

**Score the**
**Item**

- Repeat these
- Steps for each
- Criteria Item

- Complete
- Scorebook

- Finalize Key
- Themes

- Version 5.0    April 2008

- File 1 of 3

## Scoring Aspects – A-D-L-I

- Approach (A)
- Method used to accomplish a process or system
- Covers applicable aspects of criteria
- Approached is planned
- Method used is systematic or repeatable
- Deployment (D)
- Extent process is disseminated throughout the organization (depth and breath)
- Approach is deployed consistently in all areas of the organization

- Version 5.0    April 2008

- File 1 of 3

## Scoring Aspects – A-D-L-I

- Learning (L)
- Cycles of evaluation and improvement
- Encourages breakthrough change to approaches
- Sharing, adapting and adopting refinements, and innovations within the organization
- Integration (I)
- Approach aligned with organizational needs (below 65%)
- Approach integrated between organizational units (above 65%)
- Harmonization of plans, processes, and results to support organization-wide goals
- Individual components operate as a fully interconnected unit

- Version 5.0    April 2008

- File 1 of 3

## Reacting to the Problem
General Improvement Orientation
Systematic Evaluation and Improvement
Learning and Strategic Improvement
Organizational Analysis and Innovation

- Reacting to the Problem
- General Improvement Orientation
- Systematic Evaluation and Improvement
- Learning and Strategic Improvement
- Organizational Analysis and Innovation

- Version 5.0    April 2008

- File 1 of 3

## Levels of Maturity - Basic, Overall, Multiple Requirements

_**Basic**__ Item requirements_
- Responds to most basic or fundamental theme of the Item
_**Overall**__ Item requirements_
- Responds to the central or most significant features or theme
_**Multiple**__ Item requirements_
- Responds to the detailed requirements of the Item

- Version 5.0    April 2008

- File 1 of 3

## Slide 40

- Version 5.0    April 2008

- File 1 of 3

## Examining an Item

- The Criteria & Criteria Reference
- Two Types of Questions: How and What?
- Key Factors
- The Response & Level of Maturity
- Two Types of Comments: Strength and OFI
- Scoring Aspects: A-D-L-I

- Version 5.0    April 2008

- File 1 of 3

## Strength Comments

- Subject from the Criteria or application
- The senior leaders
- The organization
- The school
- The medical center
- The cemetery
- The applicant

- Verb(s) and specific Requirements from the Criteria
- review organizational performance
- Communicates, listens, and learns
- Makes needed data and information available

- Version 5.0    April 2008

- File 1 of 3

## EXAMPLE
1.1.a(1)  (A/D) Senior leaders set and deploy the organization’s vision and values through the leadership system to the workforce...

Notes: 
Address – Item 1.1, Area “a”, multiple requirement (1)
(A/D) indicates that this strength is primarily related to APPROACH and DEPLOYMENT. 
When you pull from the Criteria to write your comment, you must be accurate.  The Criteria says “to the workforce, to key suppliers and partners, to patients and to other customers”.  In this case – there is the link to the Criteria and that is what is evidenced in the application.

- EXAMPLE
- 1.1.a(1)  (A/D) Senior leaders _set and deploy the organization’s vision and values through the leadership system to the workforce..._
- Notes:
- Address – Item 1.1, Area “a”, multiple requirement (1)
- (A/D) indicates that this strength is primarily related to APPROACH and DEPLOYMENT.
- When you pull from the Criteria to write your comment, you must be accurate.  The Criteria says “to the workforce, to key suppliers and partners, to patients and to other customers”.  In this case – there is the link to the Criteria and that is what is evidenced in the application.

- Strength Comments

- Version 5.0    April 2008

- File 1 of 3

## Transition
By/through/using

This is demonstrated by 

- By/through/using
- This is demonstrated by

- Example(s) from the application
- Provide the activities or processes that led you to the conclusion that this was a strength – the facts, the WHAT, the HOW

- Strength Comments

- Version 5.0    April 2008

- File 1 of 3

## Example:
...by using an annual Strategic Planning process and numerous communication methods, such as news letters, meetings, and stakeholder sessions to communicate to all faculty, staff, students, and key stakeholders and partners.

Note:  
“By using” is the transitional phrase
The rest of the statement comes directly from the application. 

- Example:
- ...by using **an annual Strategic Planning process and numerous communication methods, such as news letters, meetings, and stakeholder sessions to communicate to all faculty, staff, students, and key stakeholders and partners.**
- Note:
- “By using” is the transitional phrase
- The rest of the statement comes directly from the application.

- Strength Comments

- Version 5.0    April 2008

- File 1 of 3

## Category 3Focus on Customers and Markets 

- Item 3.1 Customer & Market Knowledge
- Customer and Market Knowledge
- Item 3.2 Customer Relationships and Satisfaction
- Customer relationship building
- Customer satisfaction determination

- Version 5.0    April 2008

- File 1 of 3

## The applicant uses the voice-of-the-customer information and feedback to become more patient-focused and to identify opportunities for innovation through execution of two processes: (1) “walking the walk,” and (2) incorporating information learned into its Executive Board, strategic planning activities (Figure 2.1-1), and resultant goals and action plans (Figure 2.1-3). 

- The applicant _uses the voice-of-the-customer information and feedback to become more patient-focused and to identify opportunities for innovation_ through **execution of two processes: (1) “walking the walk,” and (2) incorporating information learned into its Executive Board, strategic planning activities (Figure 2.1-1), and resultant goals and action plans (Figure 2.1-3).**

- Strength Comment Example 3.1a(3)

- Version 5.0    April 2008

- File 1 of 3

## EXERCISE - Item 3.1

- Read Item 3.1 from the Criteria
- Identify Key Factors that go with Item 3.1
- Read case response to Item 3.1
- Prepare a STRENGTH COMMENT and record it on chart paper
- _Underline link to Criteria_
- Label Comments: Write the item address, the key factor numbers, and the appropriate scoring aspects

- Circle link to application

- Version 5.0    April 2008

- File 1 of 3

## Carey Examination Process
Conflict of Interest & Core Values
Baldrige Framework and Criteria Book
Organizational Profile and Key Factors
Process Scoring Aspects and Item Requirements
Category 3 and Exercises
Strength Comments

- Carey Examination Process
- Conflict of Interest & Core Values
- Baldrige Framework and Criteria Book
- Organizational Profile and Key Factors
- Process Scoring Aspects and Item Requirements
- Category 3 and Exercises
- Strength Comments

- Version 5.0    April 2008

- File 1 of 3

## Feedback 

- What went well?
- What can we do better?
- What was the most important thing you learned today?
- Based on today’s topics, what questions remain unanswered?

- Version 5.0    April 2008

- File 1 of 3