# Presentation: 562948

## Carey Application Writingand Examiner Training2008

- _Target audience_:	Analysts, managers, and quality professionals 		planning to develop a Baldrige-based 			application or join the Carey Board of 			Examiners.
- _Type of training_:	Classroom
- _Duration_:	3 days

- Version 1.3

- File 1 of 6

## Purpose of Course

- Provide VA organizations with the skills needed to produce a quality Carey application.
- Provide participants with a working knowledge of the 2008 Baldrige Criteria and how to apply it within an organization as part of an assessment and improvement process.
- Provide participants with basic skills needed to participate in the examination of a 2008 Carey evaluation process.  This includes:
- General understanding of the Criteria
- Identification and drafting of key Strengths
- Identification and drafting of Opportunities for Improvement
- Identification and drafting of scores associated with Process and Result Items

- Version 1.3

- File 1 of 6

## Course Objectives

- Participants will be able to:
- Draft the Organizational Profile section of a Carey/Baldrige application.
- Draft responses to Process and Results Baldrige Criteria questions.
- Outline and describe the Carey examination process steps, including:
- Development of a Key Factors Worksheet
- Drafting Strength comments
- Drafting Opportunity for Improvement comments
- Scoring application Items using the Baldrige Scoring guidelines

- Version 1.3

- File 1 of 6

## General Information

- Emergency telephone number
- Messages
- Rest Rooms
- Food
- Start/stop times
- Questions???

- Version 1.3

- File 1 of 6

**Notes:**
-- If you should have an emergency situation while you are here, please get the help that you need.  If possible, please also contact Eric Malloy, the Carey Program Manager. His cell phone number is 703-606-0654.  That way we know what happened to you.

## Ground Rules

- Respect opinions of others
- All questions are good questions
- What is said in class – stays in class
- Back from breaks timely
- Work hard and have FUN
- Value the diversity of group members
- Listen and have an open mind
- Others?

- Version 1.3

- File 1 of 6

## Carey Office and Instructors

- Carey Program Office (008B3)
- Scott Holliday, Director (Management Systems Improvement Service 008B3)
- Eric Malloy
- Gwendolyn Young
- Patrick Hyousse
- Other Instructors
- Kurt Gundacker, VA Systems Consultant, EES

- Version 1.3

- File 1 of 6

**Notes:**
-- Scott Holliday is our Director.
-- Eric Malloy is the Program Manager.  Please take time to introduce yourself while you are here.  He would like to meet you.  To help you break the ice with him, Eric will be here on day 2 to teach a session with you in the morning.
-- Tom Garin: First became involved with the Baldrige Program in 1990 at an ASQ Conference. Frederick Smith, Federal Express, was our speaker and told how he changed on what he learned about customers. In 1997, I received my first formal training on the Performance Excellence Criteria. I began working for the Carey Program Office in March 2006.  I am also a member of the Board of Examiners for the State of Maryland.
-- We are fortunate to have Joanne Puckett, WRJ, with us today. We will be using the WRJ Case Study during these next few days. Joanne has had a big part in producing the WRJ application and she can speak with authority when we discuss the case study. At this time, I will let Joanne introduce herself to you. 
-- Thank you Joanne. Our last instructor is Maureen Washburn, VISN 17.  Some of you probably know her.  She will be working primarily in the room across the hall with Eric.  She too is available to assist you.  Please introduce yourself to her during one of the breaks. 

## Pre-Test

- Purpose of this test is to determine subject matter knowledge before training begins.
- Individual test scores not given to your supervisor.
- Name is optional.
- 25 questions, 4 points each
- No penalty for guessing
- 10 minutes – maximum
- Each test has an identifier code at the top of the first page.  If you want to know your score – ask the instructor (Scores are associated with codes).

- Version 1.3

- File 1 of 6

**Notes:**
-- Our first order of business is the pre-test. Please work on your own and don’t reference any other material. This is a closed-book test. It is especially useful to us in assessing our teaching effectiveness.  So please take your time and give it serious consideration. I will collect the test when you are done and Eric will grade all of the tests this evening. Your name is optional, but if you want to know how you did, then we need to know your name so we can tell you. If you finish early, please feel free to move around. I ask that you not speak while in the classroom so those still taking the test can focus on their work. We will come back together at (blank) time and start the next section which class introductions.
-- Allow 20 minutes for the test

## Introductions

- Total Time – 5 minutes
- Pick a partner and gather the following information
- Name
- Organization
- Baldrige or Carey criteria experience
- Something others might not know about them
- Introduce your partner to the class
- Approximately one  minute

- Version 1.3

- File 1 of 6

**Notes:**
-- Thank you for completing the test and getting back to your seats on time so we can start.
-- We will now take a few moments to introduce ourselves to each other.  I know that some of you already know others in the class, but some people may not (like me).  Please give everyone your attention when they are speaking. Please pick a partner.  If there is someone in the room that you have not met before, this is a great time to go over and introduce yourself.  Please feel free to change your seats or move your chairs. After about five minutes, we will get back together and begin the classroom introductions. Since we have so many people to introduce, we are asking you to try to keep introduction to a minute. Thanks for your cooperation.

## Course Agenda

- Pre-work
- Review 2008 Baldrige Criteria
- Draft an Organizational Profile and Key Factors Worksheet
- Review Key Processes and Data/Information tables document
- Review information on _[www.Baldrige21.com](http://www.baldrige21.com/)_
- Bring business plan and key measures information to class

- Version 1.3

- File 1 of 6

## Course Agenda

- Classroom training (application focus)		**Day 1**
- Learn Criteria
- Develop Organizational Profile
- Develop Key Factors Worksheet
- Understand what examiners are looking for:
- Approach (A), Deployment (D), Learning (L), Integration (I), Levels (Le), Trends (T), Comparisons (C), and Integration (I)
- Develop key high level PROCESS descriptions (Organizational Profile and Process Categories)

- Version 1.3

- File 1 of 6

## Course Agenda

- Classroom training (Application focus)		**Day 2**
- Develop key high level data/information TABLES (Organizational Profile and Process Categories)
- Respond to Criteria Questions
- Link to key processes and data/information tables (refine processes and data tables as needed)
- Identify needed results data

- Version 1.3

- File 1 of 6

## Course Agenda

- Classroom training 				**Day 3**
- (Application and Examination focus)
- Review days 1 and 2
- Develop result graphs, charts, and tables
- General examination process
- Process scoring aspects: Approach (A), Deployment (D), Learning (L), and Integration (I)
- Results scoring aspects: Levels (Le), Trends (T), Comparisons (C), and Integration (I)

- Version 1.3

- File 1 of 6

## Course Agenda

- Classroom training 				**Day 3**
- (Application and Examination focus)
- Process Strength comments
- Process Opportunity for Improvement comments
- Results Strength comments
- Results Opportunity for Improvement comments
- Scoring Process Items
- Scoring Results Items
- Linkage Between Application Writing and Examination

- Version 1.3

- File 1 of 6

## Describing the SYSTEM

- In this course we will build an application by drafting:
- A Key Factors Worksheet
- Key processes and Data/information tables
- Narrative responses to the criteria that link the key factors, processes, and tables to each other and  results

- Version 1.3

- File 1 of 6

## SPEED

- We will move through exercises quickly.
- Get a quick draft
- Label work with title and figure number
- Keep draft products in order
- The pieces all start coming together at the end of day two

- Version 1.3

- File 1 of 6

## Process for Developing an Application

- 1.  Learn Criteria
- 2.  Develop Organizational Profile
- 3.  Develop Key Factors Worksheet
- 4.  Draft Key PROCESS descriptions and data/information TABLES
- 5.  Label Draft Processes and Tables (Figure x.x-x or Table x.x-x)
- 6.  Respond to Criteria Questions
- Link to key processes and data tables (refine processes and data tables as needed)
- 7.  Identify needed results data – incorporate into tables, as appropriate

- Version 1.3

- File 1 of 6

## Process for Developing an Application

- 8.  Develop Results graphs and charts
- 9.  Review overall document
- 10.  Revision, Buy-in, and Approval of application
- 	(Application should be an accurate description of the organization that leaders and staff know and understand)
- 11.  Follow application requirements/process outlined in the 2008 Carey Application Book to apply to the award program

- Version 1.3

- File 1 of 6

## Learn Criteria (Pre-work Review)

- Walk through the 2008 Criteria Book
- Core Values and Concepts
- Framework for Performance Excellence
- Item Listing (with associated point values)
- Organizational Profile and Criteria
- Basic
- Overall
- Multiple Requirements

- Version 1.3

- File 1 of 6

## Slide 19

- Version 1.3

- File 1 of 6

**Notes:**
-- Review the Requirements: Basic, Overall, and Multiple
-- Review the Item number, title, and point value
-- Review the areas to address
-- Review Notes on the bottom of the page.  Make sure everyone knows that there is another section of notes on pages 37 to 51.

## Learn Criteria (Pre-work Review)

- Walk through the 2008 Criteria Book
- Notes
- Category and Item Descriptions
- Scoring Guidelines

- Version 1.3

- File 1 of 6

## Develop Organizational Profile (Pre-work Review)

- You should have this completed.
- The Profile is the basis for making the Criteria specific to your organization and for examiners as they work to relate the Criteria to your organization.
- The Leadership team needs to know what is in the Profile and agree with it.  If an applicant can not come to agreement on the Profile then that is a good place to begin your performance improvement journey.
- If leadership agrees with Profile responses – communicate to workforce and begin responding to Criteria questions.

- Version 1.3

- File 1 of 6

## Develop Key Factors Worksheet(Pre-work Review)

- Key Factors Worksheet
- Begin with a simple bulleted version of the Organizational Profile
- This becomes the key document for coordinating the development of the Key Processes, Key Data/Information Tables, and responses to the Criteria.
*If you do not have a Key Factors Worksheet for your organization, I encourage your team to produce one during: break, lunch, or this evening.*

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Process Items: A, D, L, I

- _Approach (A)_ refers to the methods used by an organization to address Baldrige Criteria.  Approach includes the appropriateness of the methods to the Item requirements and the effectiveness of their use.
_**Process**_ is a linked set of activities with the purpose of producing a product or service.  Processes usually have a repeatable set of steps.
_**System**_ – is a set of processes working together to satisfy a need.  For example, a communication system might include 6 to 8 communication processes (newsletters, town hall meetings, e-mail, open-door policy, web page, lunch with the director, and staff meetings) which accomplish the need to effectively communicate to the workforce, customers, partners and suppliers.

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Process Items: A, D, L, I

- _Deployment (D)_ refers to the extent to which an approach is applied in addressing the requirements of a Baldrige Criteria Item.  Deployment is evaluated on the basis of the breadth and depth of application of the approach to relevant work units throughout the organization.
- For example, an organization has a complaint management process that is used in the inpatient areas of a medical facility.  Data from this process are aggregated, analyzed, and shared with senior leaders on a quarterly basis.  This information is then used to support high-level decisions that impact future customers.
*	**The approach appears to be deployed to the inpatient areas with measurement and analysis used by senior leaders.  However, there is no evidence that the approach is deployed to other areas such as outpatient and long-term care, or that performance data are used for systematic improvements by front-line staff and their managers.*

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Process Items: A, D, L, I

- _Learning (L)_ refers to new knowledge or skills acquired through evaluation, study, experience, and innovation.  There are two distinct kinds of learning: organizational and personal.
- _Organizational learning_ is achieved through research and development, evaluation and improvement cycles, workforce and stakeholder ideas and input, best-practice sharing, and benchmarking.
- _Personal learning_ is achieved through education, training, and developmental opportunities that further individual growth.

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Process Items: A, D, L, I

- _Integration (I)_ refers to harmonization of plans, processes, information, resource decisions, actions, results, and analyses to support key organization-wide goals.
- _Alignment_ is often indicated when there is a common set of goals or objectives that functions or service lines work towards achieving.  For example, goal is to reduce costs so each functional area sponsors two cost savings initiatives.  Often associated with activity, functional, or output measures.
- _Integration_ is often indicated when there is a common set of goals or objectives that are achieved through partnerships to maximize the benefits achieved.  For example, goal is to reduce costs so three cross-functional teams are sponsored to: reduce ordering and inventory costs across the organization; reduce utility costs through more efficient lighting options; and leveraging technology to reduce training costs.   Often associated with cross-functional or outcome measures.

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Process Items: A, D, L, I

- Please turn to the Scoring Guidelines in your Criteria book
- Approach at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%
- Deployment at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%
- Learning at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%
- Integration at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%

- Version 1.3

- File 1 of 6

**Notes:**
Discuss the full range of scores for “Approach”.  

Discuss the full range of scores for “Deployment”

Discuss the full range of scores for “Learning”

Discuss the full range of scores for “Integration”

Discuss the relationship among the factors – in general, examiners begin by assessing “Approach” then go to Deployment, learning and Finally Integration.  The four scoring aspects may be in different ranges for an Item.  However, the spread of ranges for a particular Item usually does not exceed three.

## Understand what examiners are looking for in Results Items: Le, T, C, I

- _Levels (Le)_ refers to numerical information that places or positions an organization’s results and performance on a meaningful measurement scale.  Performance levels permit evaluation relative to past performance, projections, goals, and appropriate comparisons.
- For example, employee satisfaction level is 85%.  Often it is difficult to assess if levels are good or not when comparison data are not provided.  However, if levels approach 100% or 0%, depending on the type of measure, many examiners don’t need comparison data to assess performance level.

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Results Items: Le, T, C, I

- _Trends (T)_ refers to numerical information that shows the direction and rate of change for an organization’s results.  A minimum of three historical (not projected) data points is generally needed to begin to ascertain a trend.  More data points may be needed to define a statistically valid trend.
- Do data points have to represent three years?  No.  If a particular result is normally monitored and reported quarterly then 3 quarters is sufficient.

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Results Items: Le, T, C, I

- Comparisons (C) refer to performance relative to appropriate competitors, industry comparisons and/or benchmarks (within the industry or outside).
- Typical types of comparisons include: VA average; best in VA class; best in Network, Regional, or Area; industry average; best in industry; and Benchmark

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Results Items: Le, T, C, I

- _Integration (I)_ refers to the extent to which results are linked to areas that the applicant has indicated as important, either in the Organizational Profile or in response to Criteria questions in Categories 1 through 6.
- For example, the application has identified 2 customer groups (inpatient and outpatient) and 3 key requirements (access, quality of care, and education).  Customer related results should address these two groups and each of the three key requirements.
- 	If the applicant provides customer results for universities, visitors to the VACO store, and visitors to the cafeteria, then it would not be clear how those results relate to the key customer groups and key requirements that the applicant identified as important.

- Version 1.3

- File 1 of 6

## Understand what examiners are looking for in Results Items: Le, T, C, I

- Please turn to the Scoring Guidelines in your Criteria book
- Levels at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%
- Trends at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%
- Comparisons at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%
- Linkages at
- 	0 – 5%, 10 – 25%, 30 – 45%, 50 – 65%, 70 – 85%

- Version 1.3

- File 1 of 6

## Develop key high level PROCESS descriptions 

- There are many possible processes that you may want to include in your application.
- We will cover several of the key, or more commonly described, processes that should be considered when developing an application.
- An applicant should be able to refer to these processes when responding to specific criteria questions.
- Key processes identified in this course were selected through review of winning Carey applications and consensus of course development planning committee members.  Committee members included experienced examiners and application writers from across VA

- Version 1.3

- File 1 of 6

## Key Processes – for preparing an Application (Summary)

- Item 1.1 Senior Leadership
- System - Leadership map
- Item 1.2 Governance and Social Responsibility
- Table or Process - Selection and linkage of Key Communities to business (chart or table)
- Item 2.1 Strategy Development
- Process - Strategic Planning
- Item 2.2 Strategy Deployment
- Process (may be incorporated in Strategic Planning process) Action plan development process

- Version 1.3

- File 1 of 6

## Key Processes – for preparing an Application (Summary)

- Item 3.1 Customer and Market Knowledge
- Process - Identifying customers, groups, and market segments
- Process - Using customer data (from various listening posts) to determine requirements, needs, and expectations and become more customer focused
- Item 3.2 Customer Relationships and Satisfaction
- Process - Complaint Management
- Process - Customer satisfaction and dissatisfaction determination

- Version 1.3

- File 1 of 6

## Key Processes – for preparing an Application (Summary)

- Item 4.1 Measurement, Analysis, and Improvement of Organizational Performance
- Process - Selecting, collecting, and analyzing measures
- Item 4.2 Management of Information, Information Technology, and Knowledge
- Process or table - How you manage: collection and transfer of knowledge; identification, sharing, and implementation of best practices; and transfer of relevant knowledge for use in strategic planning process
- Item 5.1 Workforce Engagement
- Process - How performance management system supports high performance and workforce engagement

- Version 1.3

- File 1 of 6

## Key Processes – for preparing an Application (Summary)

- Item 5.2 Workforce Environment
- Process - to recruit, hire, place, and retain new staff.
- Item 6.1 Work Systems Design
- Process - Determining Core Competencies
- Process – Designing processes (could include improvement and design)

- Version 1.3

- File 1 of 6