# Presentation: 213576

## Carey Advanced Examiner Training2008

- Completing Result Items in an IR scorebook
- Drafting Process Item comments, providing feedback, and revising
- Drafting Result Item comments, providing feedback, and revising

- File 3 of 4     Version 1.5     March 2008

## Completing an IR scorebook – Results Category

- Transferring the facts from the application notes to a scorebook that can be shared with team members

- File 3 of 4     Version 1.5     March 2008

## COE Examination Process

- Start – COE examiner
- selection

- End

- Part I
- Receive materials,
- fax forms, read

- Part II
- Key Factors worksheet

- Part III
- IR notes and scorebook

- Part IV
- Item leads draft comments,
- share, revise

- Part V
- Consensus report

- Part VI
- Site visit, finalize feedback
- and score

- File 3 of 4     Version 1.5     March 2008

## Completing the IR scorebook

- Process for documenting Results within the IR scorebook:
- The 4 – 6 Key Factors that relate to the Item should already be recorded in the IR scorebook
- The estimated number of results should also be recorded in the IR scorebook
- Read the criteria Multiple Requirements
- In the first column of the worksheet table, “Item ref.” record the Multiple Requirement address
- In the second column, “Summary of Criteria Requirement”, insert the key point(s) from the Multiple Requirement
- Note: in Category 7 most Items only have one, two or three Multiple Requirements
- In the third column, “Relevant KF”, record the numbers associated with the key factors from the top of the page that you believe relate to this specific multiple requirement

- File 3 of 4     Version 1.5     March 2008

## Completing the IR scorebook

- Process for documenting within the IR scorebook:
- In the forth column, “Level and trends (timeframe, direction, rate of change), record facts from the application.
- Note: do not describe each chart.  Group charts logically and make factual statements (Key Customer group “X”, results for three key requirements show good levels and improving trends in two of three requirements (Figures 7.2-1, 2, and 3)
- In the fifth column, “Comparisons used and appropriateness” record appropriate facts
- Note: for the group described in the previous column, what types of comparisons are shown and do these comparisons agree with types identified in P.2.a.3 and processes in 4.1.a.2?

- File 3 of 4     Version 1.5     March 2008

## Completing the IR scorebook

- Process for documenting within the IR scorebook:
- In the sixth column, “Linkage to key segments/groups, process requirements, action plan requirements” identify and site the link from the result s shown (group identified in column 4) to the Organizational Profile or a process description from the application
- Repeat for each result or group of results
- Repeat these steps for each Multiple Requirement

- File 3 of 4     Version 1.5     March 2008

## Independent Review Worksheet- Result Item Example

- Independent Review Worksheet  **Item 7.2:**

| 
4-6 Most significant key factors (KFs) for this Item and expected number of results |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 
Customer groups – patients, patient families, inpatients, outpatients
Key Requirements – access, courtesy, quality of care
Retention rates important |  |  |  | 
Growth rate
Referral business key to future success 
Estimated # of results  (4 = overall and sat. by req.; 12 segment results; 3X growth, referral, dissatisfaction, relationship = 12) total estimate = 28 results   |  |  |
| Item ref. | Key results—include figure references | Relevant KF | Level and trends |  | Comparisons  | Linkage to key segments/groups,  |
| .a.1 | Key levels and trends in satisfaction, dissatisfaction.  Comparisons to similar organizations. | 1, 2 | Figure 7.2-1 Overall Satisfaction shows levels increasing from 76% in 2004 to 84% in 2007.
Fig. 7.2-2 – 5, inpatient in each key req. and overall.  Levels consistently good, 3 of 4 charts show positive trends. |  | Fig. 7.2-1 performance better than VHA average, peer group average and industry average.
Fig. 7.2-2 – 5, only #3 and 5 shown comparisons.  Comparisons are limited to VHA average. | 7.2-1 Linked to objective and measure in Cat. 2
Fig. 7.2-2 – 5, results for inpatient in each key req. and overall.   |
|  |  |  |  |  |  |  |

- File 3 of 4     Version 1.5     March 2008

## Completing the IR scorebook

- Process for documenting within the IR scorebook:
- Once results to all Multiple Requirements have been evaluated, it is time to score the Item
- Read across all descriptions for the First Element (Levels/Trends)
- Select the score range that best fits the performance of the applicant (indicate with a L (low), M (medium), H (high) where within the range you believe the applicant’s performance is)
- Repeat for Second Element (Trends/sustained results), Third Element (Comparisons/benchmarks), and Forth Element (Importance of Results)
- Review scores you estimated for each element then select an overall score range.  Record at the bottom of the page
- Finally, select an overall score and record to the right of the selected score range
- _Repeat for all result Items_

- File 3 of 4     Version 1.5     March 2008

## Independent Review Worksheet- Result Item Example

- 7.2 Scoring

| Guidelines | 0 – 5% | 10 – 25% | 30 – 45% | 50 – 65% | 70 – 85% | 90 – 100% |
| --- | --- | --- | --- | --- | --- | --- |
| First Element
(Levels) | Narratives | From  | scoring

M | Guidelines |  |  |
| Second Element
(Trends) |  |  | 

M | 

 |  |  |
| Third Element
(Comparisons) |  | 
L |  |  |  |  |
| Fourth Element
(Integration) |  | 
 |  | 
M |  |  |

- Range: 30 – 45%			Item 7.2 Score 40%

- File 3 of 4     Version 1.5     March 2008

## Exercise – Completing the IR scorebook

- Exercise:		  **Individual****	****	**	20 minutes
- Everyone has already evaluated the results in 7.2 of the case study
- Review the criteria
- On the IR worksheet - Complete columns one, two, and three
- Review applicant results and your notes
- Record Le, T, C, I related facts in columns 4, 5, and 6
- Repeat process until results and Criteria Multiple Requirements are addressed
- If you expected to see a specific type of result based on the Criteria Requirement, Organizational Profile responses, or Process Item responses then identify gaps under the last column (“Linkage to key segments/groups process requirements, action plan requirements
- Note all comments that represent Opportunities For Improvement by highlighting in yellow
- Complete transferring key notes and identifying any gaps
- Score Item

- File 3 of 4     Version 1.5     March 2008

## Independent Review Scorebook

- You should now know how to:
**Takes Process and Results notes in an application**
**Transfer notes and facts from an application to an IR Scorebook**
**Score Process and Result Items in an IR Scorebook**

- File 3 of 4     Version 1.5     March 2008

## Drafting Process Item comments 

- Drafting comments, providing feedback, making revisions

- File 3 of 4     Version 1.5     March 2008

## COE Examination Process

- Start – COE examiner
- selection

- End

- Part I
- Receive materials,
- fax forms, read

- Part II
- Key Factors worksheet

- Part III
- IR notes and scorebook

- Part IV
- Item leads draft comments,
- share, revise

- Part V
- Consensus report

- Part VI
- Site visit, finalize feedback
- and score

- File 3 of 4     Version 1.5     March 2008

## Drafting process comments from multiple IR scorebooks

- Basic process
- Share IR scorebooks with members of the team and provide a copy to the Carey Program
- Print a hard copy of your own IR scorebook
- For the Items that you are designated as lead, print hard copies of your teammates’ materials
- Align the hard copies of the scorebooks such that you can see responses across each Multiple Requirement
- For example:  IR worksheets showing 1.1.a.1 for everyone on the team

- File 3 of 4     Version 1.5     March 2008

## Drafting process comments from multiple IR scorebooks

- Basic process Continued
- Review the Criteria summary in column 2 and the facts in the remaining columns to the right
- Look for common facts across the sheets.  Are there supporting facts and highlighted opportunities in common?
- Review all the facts across all people for the entire Item
- Identify what appears to be several key strengths and several key opportunities
- Review the related Multiple Requirements from the criteria
- Begin drafting Strength and OFI comments for the Item in a Carey feedback report format
- See Carey Feedback Report Format

- File 3 of 4     Version 1.5     March 2008

## Carey Feedback Report Format 

***Leadership***
***1.1 Senior Leadership:  Criteria written in bold print***
**STRENGTHS**
**(1 – 5 comments)**
**OPPORTUNITIES FOR IMPROVEMENT**
***(1 – 3 comments)***
***Performance Score Range******	******	******	******	******11% - 11%***

- File 3 of 4     Version 1.5     March 2008

## Carey Feedback Report RULES

***Comment Rules:***
***At least one Strength comment unless score is 0%***
***At least one OFI comment unless score is 100%***
***Total comments per Item, 4 to 8  ***
***Try for 6 or 7 comments per Item***

- File 3 of 4     Version 1.5     March 2008

## Process Comment Format 

- Review

- File 3 of 4     Version 1.5     March 2008

## Structure – Strength Comments, Part One

- Subject from the Criteria or application
- The senior leaders
- The organization
- The school
- The medical center
- The cemetery
- The applicant

- Verb(s) and specific Requirements from the Criteria
- Review organizational performance
- Communicates, listens, and learns
- Makes needed data and information available

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- I highly recommend that everyone use the formula we are about ready to discuss.  This makes it clear to the examiner team at consensus because everyone expects what is coming and you don’t have to work very hard to find it.  It is also good feedback to the applicant because they too are familiar with the formula and won’t be surprised when they read it. They know what is coming by the way it is structured. 
-- Choose an appropriate subject and use it to begin the comment.  You do this to hide the identity of the applicant in the examination process. 
-- The next phrase should be a direct quote from the criteria so that everyone is clear on which question is being addressed. In the consensus meeting, I would highly encourage you to select someone on the team to be the criteria cop and make sure the criteria is addressed in the comment.

## Structure – Strength Comments, Part One

- EXAMPLE
- 1.1.a(1)  Senior leaders _set and deploy the organization’s vision and values through the leadership system to the workforce..._
- Note: When you pull from the Criteria to write your comment, you must be accurate.  The Criteria says “to the workforce, to key suppliers and partners, to patients and to other customers”.  In this case – there is the link to the Criteria and that is what is evidenced in the application

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- The criteria says “HOW do SENIOR LEADERS DEPLOY your organization’s VISION and VALUES through your LEADERSHIP SYSTEM, to the WORKFORCE, to KEY suppliers and PARTNERS, to PATIENTS and other CUSTOMERS, and to other STAKEHOLDERS, as appropriate?” (p. 15) 

## Structure – Strength Comments, Part One

- EXAMPLE
- 1.1.a(1)  (A/D) Senior leaders _set and deploy the organization’s vision and values through the leadership system to the workforce..._
- Note:
- Address – Item 1.1, Area “a”, multiple requirement (1)
- (A/D) indicates that this strength is primarily related to APPROACH and DEPLOYMENT

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- The address helps everyone to know which item you are referring to. Get in the habit of putting the address before each comment. In this example, the comment is written to the multiple level.
-- Given A/D/L/I, this strength comment refers to the “A” or approach and “D” or deployment.  Keep this in mind when you are generating other comments or scoring the item. For example, if you were to generate an OFI comment for 1.1.a(1) and your OFI comment was specifically related to the approach, you may have two conflicting comments that need to be resolved.  One of the comments will likely be removed.  
-- If you were scoring the item, this strength comment would not be as helpful to the applicant as a strength comment written to the multiple requirements that refers to A/D/L/I.

## Transition
By/through/using

This is demonstrated by 

- By/through/using
- This is demonstrated by

- Example(s) from the application
- Provide the activities or processes that led you to the conclusion that this was a strength – the facts, the WHAT, the HOW

- Structure – Strength Comments, Part Two

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- A “transition” is a word or a phrase that takes the reader from the first part of the comment and leads him/her to the second part of the comment. Some common words or phrases are listed on this chart.
-- The example comes directly from the application. Use enough of the writing so the applicant knows what you are talking about.  You don’t need to re-write the entire example.  Not all of the examples written in the application are used in the comments.  When writing the application, you should choose which examples to include in your application very carefully.  

## Structure - OFI Comments – Part One

- Introductory Statement
- It is not clear how/what
- It is unclear how/what
- It is not evident how/what
- It is not apparent how/what

- Subject from the Criteria or Application
- The senior leaders
- The organization’s
- The medical center
- The MSN
- The regional office

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- Prior to the site visit, each OFI comment will have an introductory statement saying “it is not clear...”  The subject of the comment usually comes from the application (for example, “the applicant”), but can also come from the criteria.

## Structure - OFI Comments – Part One (continued)

- Verb(s) and specific requirements from the Criteria
- Review organizational performance
- Communicates, listens, and learns
- Makes needed data and information available
- Complaint management process

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- The verb (or other descriptive language) usually comes from the criteria.
-- For example:
-- “Although the strategic challenge is identified, it is not clear how the applicant develops their strategic challenges.”  In this example, the examiner is telling the applicant that he/she acknowledges the fact that they have strategic challenges in the application.  What is not clear, is the process used to develop the key challenges. If the applicant receives a site visit, the applicant will need to explain to the site visit team how they develop strategic challenges. Once the site visit is completed, the OFI comment can be removed or changed. It can no longer remain unclear.

## Structure - OFI Comments – Part One

- Example:
- 3.2b(4)  It is not evident how the organization _keeps their approaches to determining satisfaction current with service needs and direction_.

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- In this example, the introductory statement is “it is not clear...,” the subject is “the organization,” and the descriptive language is “keeps their approaches to determining satisfaction current with service needs and direction.” 

## Structure - OFI Comments – Part Two

- Example(s) from the application
- Provide what led you to the conclusion that this was an OFI – what was missing?

- “So what” or “Impact” statement
- Clearly link comment to the Applicant’s Key Factors and/or scoring guideline language

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- The OFI comment has a second part that includes: (1) examples from the application and (2) a “so what” or “impact” statement. What is missing is linked to the criteria. What makes it relevant is linked to what they say is important.  The “so what” statement is what makes the comment relevant to the applicant. It is required with OFI comments. It is optional for strength comments.

## Structure - OFI Comments – Part Two

- Example:
- Although the Diversity Team worked with the Benefits Team to identify a method to deal with ethnic differences by providing a floating day for personnel holidays, it is not clear that the organization has a systematic method to tailor offerings to the diversity of its faculty and staff. As a result, the organization may not be able to overcome its _strategic challenge to attract and retain highly qualified employees_.

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- The introductory statement: (1) “Although...” gives the applicant credit for what they are already doing. (2) “It is not clear...” identifies the piece that is missing.
-- The subject: “the organization”
-- The verb or descriptive language: “that tailoring offerings exists”
-- Example from the application: the Diversity Team worked with the Benefits Team to identify a method to deal with ethnic differences by providing a floating day for personnel holidays, but it is not systematic
-- “So What:” may not be able to overcome its strategic challenge to attract and retain highly qualified employees

## Carey Feedback Report Format

- All Carey Examination teams will be provided with a standard feedback report format
- COE and Trophy reports have the same basic structure
- When/if a site visit takes place teams document findings within this feedback report
- Using “Caps lock”, teams document after each OFI and selected Strengths – KEEP, DELETE, MODIFY.  This is followed by evidence gathered while on site

- File 3 of 4     Version 1.5     March 2008

## Sample; Standard Carey Feedback Format

***Leadership***
***1.1 Senior Leadership:  How do your senior leaders lead? ******	******(70 pts.)***
***Describe how senior leaders guide and sustain your organization.  Describe how senior leaders communicate with your workforce and encourage high performance***
**STRENGTHS**
**OPPORTUNITIES FOR IMPROVEMENT**
***Performance Score Range******	******	******	******	******11% - 11%***

- File 3 of 4     Version 1.5     March 2008

## Sample; Standard Carey Feedback Format – Strength Comment

***Leadership***
***1.1 Senior Leadership:  How do your senior leaders lead? ******	******(70 pts.)***
***Describe how senior leaders guide and sustain your organization.  Describe how senior leaders communicate with your workforce and encourage high performance***
**STRENGTHS**
**1.1.a.1 A, D Senior leaders set and deploy the organization’s vision and values through the leadership system to the workforce as demonstrated through communication methods shown in the workforce segment of figure 1.1-2 and New Employee Orientation training.**
***STRUCTURE OF COMMENT – ITEM NUMBER, ASPECT (A,D,L,I), COMMENT***
**OPPORTUNITIES FOR IMPROVEMENT**
***Performance Score Range******	******	******	******	******11% - 11%***

- File 3 of 4     Version 1.5     March 2008

## Sample; Standard Carey Feedback Format – OFI comment

***Leadership***
***1.1 Senior Leadership:  How do your senior leaders lead? ******	******(70 pts.)***
***Describe how senior leaders guide and sustain your organization.  Describe how senior leaders communicate with your workforce and encourage high performance***
**STRENGTHS**
**OPPORTUNITIES FOR IMPROVEMENT**
***3.2.b.4  L  It is not clear how the organization keeps approaches to determining customer satisfaction current with service needs and direction.  Leaders risk determination methods that are not aligned with customer needs, expectations, or emerging segments and may not be address changes in customer communication preferences,  adversely impacting response rates.***
***Performance Score Range******	******	******	******	******11% - 11%***

- File 3 of 4     Version 1.5     March 2008

## Sample; Standard Feedback Format – OFI comment AFTER SITE VISIT

***Leadership***
***1.1 Senior Leadership:  How do your senior leaders lead? ******	******(70 pts.)***
***Describe how senior leaders guide and sustain your organization.  Describe how senior leaders communicate with your workforce and encourage high performance***
**STRENGTHS**
**OPPORTUNITIES FOR IMPROVEMENT**
***3.2.b.4  L  It is not clear how the organization keeps approaches to determining customer satisfaction current with service needs and direction.  Leaders risk determination methods that are not aligned with customer needs, expectations, or emerging segments and may not be address changes in customer communication preferences,  adversely impacting response rates.  DELETE – APPLICANT REVIEWS AND REVISES SURVEY QUESTIONS AND TARGET GROUPS ANNUALLY AND HAS MODIFIED COLLECTION METHODS TO REFLECT CUSTOMER’S PREFERENCE FOR COMPUTER-BASED SURVEYS***
***Performance Score Range******	******	******	******	******11% - 11%***

- File 3 of 4     Version 1.5     March 2008

## Process Comment 

- Exercise

- File 3 of 4     Version 1.5     March 2008

## Exercise – Drafting Process Comments

- Exercise:     **Team**			25 minutes
- Using the IR worksheets produced by your team (table):
- Review facts presented for 1.1.a.1
- Review the Criteria summary in column 2 and the facts in the remaining columns to the right
- Look for common facts across the sheets.  Are there supporting facts and highlighted opportunities in common?
- Identify what appears to be one key strength and one key opportunity
- Review the related Multiple Requirements from the criteria
- _Draft one Strength and one OFI comment_
- Record on Chart paper – underline link to Criteria, circle examples (facts) from application

- File 3 of 4     Version 1.5     March 2008

## Drafting Result Item comments 

- Drafting comments, providing feedback, making revisions

- File 3 of 4     Version 1.5     March 2008

## COE Examination Process

- Start – COE examiner
- selection

- End

- Part I
- Receive materials,
- fax forms, read

- Part II
- Key Factors worksheet

- Part III
- IR notes and scorebook

- Part IV
- Item leads draft comments,
- share, revise

- Part V
- Consensus report

- Part VI
- Site visit, finalize feedback
- and score

- File 3 of 4     Version 1.5     March 2008

## Drafting result comments from multiple IR scorebooks

- Basic process
- Share IR scorebooks with members of the team and provide a copy to the Carey Program
- Print a hard copy of your own IR scorebook
- For the Items that you are designated as lead, print hard copies of your teammates’ materials
- Align the hard copies of the scorebooks such that you can see responses across each Multiple Requirement. For example:  IR worksheets showing 7.2 for everyone on the team

- File 3 of 4     Version 1.5     March 2008

## Drafting result comments from multiple IR scorebooks

- Basic process Continued
- Review the Criteria summary in column 2 and the facts in the remaining columns to the right
- Look for common facts across the sheets.  Are there supporting facts and highlighted opportunities in common?
- Review all the facts across all people for the entire Item.
- Identify what appears to be several key strengths and several key opportunities
- Review the related multiple requirements from the criteria
- Begin drafting Strength and OFI comments for the Item in a Carey feedback report format
- See Carey Feedback Report Format

- File 3 of 4     Version 1.5     March 2008

## Results Comment Format 

- Review

- File 3 of 4     Version 1.5     March 2008

## Category 7Examination of Results

- Draft comments – Make your point (focused on one or more scoring aspects) and then use information in the application to support it
- Performance _trends_ in the key areas of “X” and “Y” are “Z” as shown in figure 7...
- Performance _comparisons_ shown for ...

- File 3 of 4     Version 1.5     March 2008

## Category 7Results Comments

- Elements: subject (applicant or result), timeframe (in, to/from), numbers (be specific), figure reference number (always cited), and so what statement (if applicable)
- All key areas of the Item responses should be reflected in the Item feedback comments
- Scoring Aspects: all scoring aspects (Le, T, C, I) should be addressed in comments

- File 3 of 4     Version 1.5     March 2008

## Category 7Example of Strength Comment – Exercise

- What scoring aspects are addressed in this strength comment?
- Overall satisfaction levels are _consistently higher than VHA average_ in each of the three _key customer requirements (access, courtesy, and quality of care_) as shown in Figure 7.2-3.  The Overall Customer Satisfaction – Access levels are in the mid-90s for the past three years, consistently above VHA average (see Figure 7.2-4).  The Overall Customer Satisfaction – Courtesy levels show an increase from 80% in 2005 to 88% in 2007 (see Figure 7.2-5).  The Overall Customer Satisfaction – Quality of Care levels are sustained at high levels of satisfaction, 97% each of the past three years (see Figure 7.2-6).  This places the applicant in the top 10 VHA facilities each year for quality of care

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- Scoring Aspects addressed in this Strength Comment:
Level -- Overall satisfaction levels are consistently higher than VHA average in each of the three key customer requirements (access, courtesy, and quality of care) as shown in Figure 7.2-3. The Overall Customer Satisfaction – Access levels are in the mid-90s for the past three years, consistently above VHA average (see Figure 7.2-4). The Overall Customer Satisfaction – Courtesy levels show an increase from 80% in 2003 to 88% in 2005 (see Figure 7.2-5). The Overall Customer Satisfaction – Quality of Care levels are sustained at high levels of satisfaction, 97% each of the past three years (see Figure 7.2-6). 
Trend -- The Overall Customer Satisfaction – Access levels are in the mid-90s for the past three years, consistently above VHA average (see Figure 7.2-4). The Overall Customer Satisfaction – Courtesy levels show an increase from 80% in 2003 to 88% in 2005 (see Figure 7.2-5). The Overall Customer Satisfaction – Quality of Care levels are sustained at high levels of satisfaction, 97% each of the past three years (see Figure 7.2-6). 
Comparison -- consistently higher than VHA average... This places the applicant in the top 10 VHA facilities each year for quality of care. 
Linkages – Not Sure
Segmentation -- key customer requirements (access, courtesy, and quality of care) 

## What scoring aspects are addressed in this OFI comment?
7.1a  While the applicant shows overall customer satisfaction with product quality for 2007, no trend data are presented to allow it to track the rate of improvement (Figure 7.1-1). In addition, the data are not segmented by key customer group (i.e., industrial or commercial). Thus, it is not clear how the applicant identifies variation in its customer satisfaction over time or from sector to sector to target opportunities for improvement

- What scoring aspects are addressed in this OFI comment?
- 7.1a  While the applicant shows overall customer satisfaction with product quality for 2007, no trend data are presented to allow it to track the rate of improvement (Figure 7.1-1). In addition, the data are not segmented by key customer group (i.e., industrial or commercial). Thus, it is not clear how the applicant identifies variation in its customer satisfaction over time or from sector to sector to target opportunities for improvement

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- Scoring Aspects addressed in this OFI comment:
Level – “Shows overall customer satisfaction with product quality for 2002” (may be present)
Trend – “No trend data”
Comparisons – Not mentioned
Linkages – Customer Satisfaction may link to key factors, but need more information
Segmentation – “Not segmented by key customer groups (i.e., industrial or commercial)”

## Category 7Example of OFI Comment – Exercise

- What scoring aspects are addressed in this OFI comment?
- Although overall satisfaction levels are shown, no segmented results are provided.  The applicant identified inpatient, elderly women, and long-term care as key customer segments.  Without results for these segments it may be difficult for leaders to assess progress in meeting the unique needs identified as important to satisfying these customers and meeting business objectives

- File 3 of 4     Version 1.5     March 2008

**Notes:**
-- Scoring Aspects addressed in this OFI Comment
Levels – “overall satisfaction levels”
Trends – n/a
Comparisons – n/a
Linkages – “overall satisfaction levels” may be important (need more information)
Segmentation – “no segmented results are provided”
Gaps – “The applicant identified in-patient, elderly women, and long-term care as key customer segments” (but no segmented results are provided). 

## Result Comments  

- Exercise

- File 3 of 4     Version 1.5     March 2008

## Exercise – Drafting Result Comments

- Exercise	**Team**			25 minutes
- Using the IR worksheets produced by your team (table):
- Review facts presented for 7.2
- Review the Criteria summary in column 2 and the facts in the remaining columns to the right
- Look for common facts across the individual worksheets.  Are there supporting facts and highlighted opportunities in common?
- Identify what appears to be one key strength and one key opportunity
- Review the related Multiple Requirements from the criteria
- _Draft one Strength and one OFI comment_
- Record on Chart paper – underline key scoring aspect, circle examples (facts) from application

- File 3 of 4     Version 1.5     March 2008

## COE Consensus 

- Process

- File 3 of 4     Version 1.5     March 2008

## COE Examination Process

- Start – COE examiner
- selection

- End

- Part I
- Receive materials,
- fax forms, read

- Part II
- Key Factors worksheet

- Part III
- IR notes and scorebook

- Part IV
- Item leads draft comments,
- share, revise

- Part V
- Consensus report

- Part VI
- Site visit, finalize feedback
- and score

- File 3 of 4     Version 1.5     March 2008

## Circle of Excellence -  Examination Process

- Part IV
- Team leader identifies Item leads
- Leads draft consensus comments and scores in Feedback report format
- Each Item is a separate file
- File naming format:
- Item, Item number, lead person’s initials, version number.  Example: Item_1_1_EM_V1  This means: Item 1.1, lead is Eric Malloy, version 1
- Item leads share V1 files with all members of the team
- All team members review and comment on all V1 files.  If no comment – then comment that V1 is OK
- When commenting on a file, team members will rename files as follows:  Item, Item number, lead person’s initials, version number, reviewer initials
- Example: Item_1_1_EM_V1_TG  This means: Item 1.1 lead is Eric Malloy, version 1, reviewed by Tom Garin
- Leads consider all V1 comments, revise, and rename file V2
- Item leads share V2 with all team members
- Team members comment on all V2 files, rename and return to Item leads

- File 3 of 4     Version 1.5     March 2008

## Circle of Excellence -  Examination Process

- Leads consider all V2 comments, revise, and rename file V3
- V3 is shared with members of the team along with a note that says, based on feedback we need to discuss “X” and “Y” via conference call to resolve.  Or – no issues to discuss during conference call
- THERE SHOULD NOT BE A LOT OF ISSUES TO DISCUSS DURING A CONSENSUS CALL
- REVIEWERS SHOULD ONLY COMMENT ON ISSUES – NOT GRAMMAR.  MOST ISSUES SHOULD HAVE BEEN ADDRESSED PRIOR TO THE CONSENSUS CALL
- Hold consensus call(s) to discuss only those issues which are not resolved
- Item leads make final revisions and provides V4 file to the team leader

- File 3 of 4     Version 1.5     March 2008

## COE Examination Process

- Start – COE examiner
- selection

- End

- Part I
- Receive materials,
- fax forms, read

- Part II
- Key Factors worksheet

- Part III
- IR notes and scorebook

- Part IV
- Item leads draft comments,
- share, revise

- Part V
- Consensus report

- Part VI
- Site visit, finalize feedback
- and score

- File 3 of 4     Version 1.5     March 2008

## Circle of Excellence -  Examination Process

- Part V
- Team leader consolidates all eighteen V4 files into one consensus report, renames the file and shares with members of the team and Carey program prior to site visit
- Standard consensus report name: COE, Team name, Consensus, Team leader initials
- Example: COE_BEAR_CONSENSUS_EM
- Part VI
- Team leader reviews standard Carey site visit agenda and team consensus report
- Drafts a COE agenda (shortened version of a standard site visit agenda) with a focus on those areas in greatest need of clarification
- Site visit agenda agreed to with team
- Shared with (revise if necessary) and agreed to with applicant

- File 3 of 4     Version 1.5     March 2008

## Circle of Excellence -  Examination Process

- Part VI
- Arrange for site visit
- Conduct site visit
- Document findings within feedback report
- Finalize feedback report and scores
- Closing meeting at site
- Send feedback report to Carey Program Manager.  Once Manager has confirmed receipt of the report, all team members are to destroy applications, all hard and soft copies of notes, versions, and any other information related to the examination of the application

- File 3 of 4     Version 1.5     March 2008

## COE and Trophy Processes

- Frequently Asked Questions

- File 3 of 4     Version 1.5     March 2008

## Circle of Excellence - FAQs

- Why make 18 files and follow this crazy naming standard?
- It is a proven way to keep track of information efficiently
- Why circulate the draft comments, revise, and then have the consensus call?  Let’s just have the call!
- If you get input and revise prior to the call – it will take far less time to reach consensus
- What is the biggest factor in the amount of time needed to reach consensus?
- If you focus on the facts and basic agreement with comments it will be quick
- If you wordsmith comments make sure you have a comfortable chair and a LOT OF TIME
- KEY POINT TO REMEMBER – ALL COE APPLICANTS GO TO SITE SO DON’T WASTE TIME WITH WORDING

- File 3 of 4     Version 1.5     March 2008

## Trophy process – the key difference is that consensus is face-to-face

- Consensus happens Face-to-Face
- Leads draft comments for assigned Items
- Teams discuss Items (leads facilitate discussions and take notes on revisions)
- Team leaders need to make sure everyone participates
- Item leads revise their Items
- Team Leaders – are usually responsible for executive summary, key themes, putting the pieces together

- File 3 of 4     Version 1.5     March 2008

## Trophy process -  FAQs

- Why did I learn the IR scorebook?
- It is highly recommended that the team leader examine the entire application.  The IR scorebook is an excellent tool for examining the entire application
- BIG REASON – the program is moving to incorporate the IR scorebook in all future examination processes
- Why do team leaders need to examine the entire package?
- Team Leader are responsible for the overall product.  There is a strong possibility that one or more people from your team are unable to complete the sections assigned.  As a result, you may need to assume their sections
- With all this extra work, why would anyone agree to be a team leader?
- Being a lead is the next level.  Helping others and sharing will increase your understanding of the criteria
- You know how important this task is and you are willing to do what it takes to help ensure that applicants receive quality feedback for the time they invested

- File 3 of 4     Version 1.5     March 2008