Step-by-Step Instructions for INDEPENDENT REVIEW Scorebook Preparation

+----------+-----------------------------------------------------------+
| ##       | The most important product you will deliver as an         |
| ### Intr | Examiner is your scorebook---your written evaluation of   |
| oduction | an Award applicant. Scorebooks ultimately end up in the   |
|          | hands of Award applicants and provide them with a         |
|          | Criteria-based assessment of their organization's         |
|          | strengths and opportunities for improvement (OFIs). These |
|          | instructions take you through a process for completing    |
|          | your Independent Review. The observations you develop at  |
|          | this stage will become inputs for a team of Examiners to  |
|          | use in developing a Consensus Scorebook.                  |
+----------+-----------------------------------------------------------+
| #####    | -   Schedule 20--40 hours to complete your scorebook.     |
| Before Y |                                                           |
| ou Begin | -   Review the materials in your evaluation mailing and   |
|          |     **complete the Conflict of Interest Form**.           |
|          |                                                           |
|          | -   From the BNQP Web site                                |
|          |     (www.baldrige.nist.gov/Examiner_Resources.htm), click |
|          |     on *Independent Review*. Download the *Independent    |
|          |     Review Scorebook* to use for your work. You will use  |
|          |     a different set of worksheets during the Consensus    |
|          |     and Site Visit Reviews.                               |
+----------+-----------------------------------------------------------+
| ####     |   -- ---------------------------------------------------  |
| # Steps  |      **Review the appropriate Criteria booklet.**         |
|          |                                                           |
| ##       |      **Read the application.**                            |
| ### (Det |                                                           |
| ails for |      **Draft an initial list of key factors.**            |
|  each st |                                                           |
| ep are o |      **(Optional) Draft initial responses to the key      |
| n the fo |      theme questions.**                                   |
| llowing  |                                                           |
| pages.)  |      **Read and begin to evaluate each Process Item in    |
|          |      Categories 1--6.**                                   |
|          |                                                           |
|          |      **Analyze the applicant's response to each Process   |
|          |      Item, and record your observations on an             |
|          |      Independent Review Worksheet.**                      |
|          |                                                           |
|          |      **Determine priority observations, and bold the      |
|          |      text on the worksheet.**                             |
|          |                                                           |
|          |      **Determine if any important Criteria requirements   |
|          |      or relevant key factors are not addressed, and       |
|          |      complete the lower section of the worksheet.**       |
|          |                                                           |
|          |      **Score the Item using the associated scoring        |
|          |      grid.**                                              |
|          |                                                           |
|          |      **Read and begin to evaluate each Results Item in    |
|          |      Category 7.**                                        |
|          |                                                           |
|          |      **Analyze the results to determine how well the      |
|          |      applicant is performing.**                           |
|          |                                                           |
|          |      **Determine priority observations, and bold the      |
|          |      text on the worksheet.**                             |
|          |                                                           |
|          |      **Determine if any important Criteria requirements   |
|          |      or relevant key factors are not addressed, and       |
|          |      complete the "Overall Item Notes" section of the     |
|          |      worksheet.**                                         |
|          |                                                           |
|          |      **Score the Item using the associated scoring        |
|          |      grid.**                                              |
|          |                                                           |
|          |      **Review all Independent Review Worksheets.**        |
|          |                                                           |
|          |      **Record all scores on the Score Summary             |
|          |      Worksheet.**                                         |
|          |                                                           |
|          |      **Finalize the key factors.**                        |
|          |                                                           |
|          |      **(Required) Finalize a bulleted list of responses   |
|          |      under a, b, c, and d on the Key Themes Worksheet.**  |
|          |                                                           |
|          |      **Record the information requested on the front      |
|          |      cover of the scorebook.**                            |
|          |                                                           |
|          |      **Assemble the scorebook pages using the prescribed  |
|          |      sequence.**                                          |
|          |                                                           |
|          |      **Upload your scorebook to the secure Web site,      |
|          |      *examinerdepot*.**                                   |
|          |   -- ---------------------------------------------------  |
+----------+-----------------------------------------------------------+

+-----------+----------------------------------------------------------+
| ###       | > Use the instructions in the following table to         |
| ## Detail | > complete your scorebook. **steps** are general;        |
| ed Steps  | > **actions** include details and notes.                 |
+-----------+----------------------------------------------------------+

+-----------+----------------------------------------------------------+
| Steps     | Actions                                                  |
+===========+==========================================================+
| **Review  | Review these sections of the appropriate Criteria        |
| the       | booklet (Business/Nonprofit, Education, or Health Care): |
| Criteria  |                                                          |
| b         | -   Core Values and Concepts                             |
| ooklet.** |                                                          |
|           | -   Criteria requirements                                |
|           |                                                          |
|           | -   Criteria Response Guidelines                         |
|           |                                                          |
|           | -   Scoring System and Scoring Guidelines                |
+-----------+----------------------------------------------------------+
| **Read    | -   **Read the entire application, highlighting and/or   |
| the       |     noting the applicant's key processes---the methods   |
| appli     |     used to address Item requirements---and noting       |
| cation.** |     results you would expect to see related to these     |
|           |     processes.**                                         |
+-----------+----------------------------------------------------------+
| **Draft   | > **Definition**: A "key factor" is an attribute of an   |
| an        | > organization or its environment that influences the    |
| initial   | > way the organization operates and the key challenges   |
| list of   | > it faces. Examiners use key factors to focus their     |
| key       | > assessments on what is important to the applicant.     |
| f         |                                                          |
| actors.** | -   Read the Criteria questions for Items P.1 and P.2.   |
|           |                                                          |
|           | -   Read the applicant's Organizational Profile, which   |
|           |     responds to the Criteria questions in Items P.1 and  |
|           |     P.2.                                                 |
|           |                                                          |
|           | -   Use these responses to draft an initial list of key  |
|           |     factors.                                             |
|           |                                                          |
|           | -   Record these key factors---listed in the order of    |
|           |     the questions in P.1 and P.2---on the Key Factors    |
|           |     Worksheet in the scorebook.                          |
|           |                                                          |
|           | -   Note that you may find other key factors in the      |
|           |     Eligibility Certification Forms or in the            |
|           |     applicant's responses to the Criteria requirements;  |
|           |     you may add these key factors at any time. The       |
|           |     development of the Key Factors Worksheet is an       |
|           |     iterative process.                                   |
|           |                                                          |
|           | > *For a sample completed Key Factors Worksheet, see the |
|           | > Examiner Learning Resource Center                      |
|           | > ([www.baldrige.nist.gov/Examiner_Resources.htm]        |
|           | (http://www.baldrige.nist.gov/Examiner_Resources.htm)).* |
+-----------+----------------------------------------------------------+
| **(       | > **Definition**: A "key theme" is a strength or OFI     |
| Optional) | > that addresses a central requirement of the Criteria,  |
| Draft an  | > is common to more than one Item or Category            |
| initial,  | > (crosscutting), is especially significant in terms of  |
| bulleted  | > the applicant's key factors, and/or addresses a Core   |
| list of   | > Value of the Criteria.                                 |
| responses |                                                          |
| to key    | -   Respond to the questions under a, b, c, and d on the |
| theme     |     Key Themes Worksheet. These questions are designed   |
| que       |     to capture the overarching observations that you     |
| stions.** |     noted when you read the entire application.          |
|           |                                                          |
|           | > *This step is optional at this point. Many Examiners   |
|           | > find it helpful to draft initial responses as a way of |
|           | > capturing their first impressions of the applicant.    |
|           | > Later in the evaluation process, **you are required to |
|           | > complete the Key Themes Worksheet.** (See the final    |
|           | > page of these instructions.)*                          |
+-----------+----------------------------------------------------------+
| **Read    | ![](media/image5.jpeg){width="2.151388888888889in"       |
| and begin | height="1.83125in"}                                      |
| to        |                                                          |
| evaluate  | > PROCESS ITEM EVALUATION STEPS                          |
| each      | >                                                        |
| Process   | > For each Item, complete the six-step Item evaluation   |
| Item in   | > process and record observations on the Independent     |
| C         | > Review Worksheet.                                      |
| ategories |                                                          |
| 1--6.**   | 1.  **Read the Criteria Item to refresh your memory and  |
|           |     understand the Item requirements.**                  |
| ![](m     |                                                          |
| edia/imag | 2.  **Determine the most relevant key factors, and       |
| e1.jpeg){ |     record them on the Independent Review Worksheet.**   |
| width="1. |                                                          |
| 101388888 |     -   Review your Key Factors Worksheet to determine   |
| 888889in" |         the attributes of the organization that would    |
| height="0 |         influence its responses to the Item              |
| .9375in"} |         requirements.                                    |
|           |                                                          |
| ![](m     |     -   Select the most relevant four to six key factors |
| edia/imag |         for the Item. These will be a subset of those    |
| e2.jpeg){ |         listed on the Key Factors Worksheet and may even |
| width="1. |         be a subset of one key factor (e.g., one         |
| 101388888 |         strategic challenge that is most relevant to the |
| 888889in" |         Item rather than the entire set of strategic     |
| height="0 |         challenges).                                     |
| .9375in"} |                                                          |
|           |     -   Record these four to six key factors and their   |
| ![](m     |         corresponding numbers in the space provided at   |
| edia/imag |         the top of the Independent Review Worksheet for  |
| e3.jpeg){ |         the Item.                                        |
| width="1. |                                                          |
| 101388888 | 3.  **Read the relevant section of the application.**    |
| 888889in" |                                                          |
| h         | 4.  **Identify the processes or approach the applicant   |
| eight="0. |     uses to meet Item requirements.**                    |
| 93125in"} |                                                          |
|           | 5.  **Flag, mark up, and/or take notes as needed.**      |
| ![](me    |                                                          |
| dia/image |     -   In noting the processes or approach, record the  |
| 4.jpeg){w |         applicant's wording from the application rather  |
| idth="1.0 |         than using your own words.                       |
| 868055555 |                                                          |
| 555556in" |     -   On the application, note any measure/indicator   |
| h         |         you expect to see reported in Category           |
| eight="0. |         7---Results.                                     |
| 93125in"} |                                                          |
|           |     -   Note any ideas, threads, or patterns that recur  |
|           |         in multiple Items or Categories. These may be    |
|           |         added to your initial set of key themes.         |
|           |                                                          |
|           | 6.  **Continue to develop the Independent Review         |
|           |     Worksheet.**                                         |
|           |                                                          |
|           |     -   Identify and note in the second column the key   |
|           |         processes, approach, or methods used by the      |
|           |         applicant (e.g., how senior leaders deploy       |
|           |         vision and values: quarterly newsletters and     |
|           |         monthly meetings).                               |
|           |                                                          |
|           |     -   In the first column, as appropriate, note the    |
|           |         specific Areas of the Item that the processes or |
|           |         methods address (e.g., 1.1\[a\]1--3).            |
|           |                                                          |
|           |     -   Identify and note in the third column the        |
|           |         numbers of relevant key factors that influence   |
|           |         this approach or process (e.g., varied           |
|           |         employees, shifts, sites).                       |
+-----------+----------------------------------------------------------+
| **Analyze | -   **In the next four columns of the Independent Review |
| the       |     Worksheet, briefly record observations under each    |
| ap        |     evaluation factor---approach, deployment, cycles of  |
| plicant's |     learning, and integration---keeping the following    |
| response  |     factors and questions in mind. Be sure to list the   |
| to each   |     *specific* evidence that you observed (e.g., instead |
| Process   |     of writing "approach is systematic," write "approach |
| Item, and |     XYZ has six steps and includes a final step for      |
| record    |     evaluation and feedback."). Mark favorable           |
| your      |     observations with a "+" sign and unfavorable         |
| obs       |     observations with a "-" sign.**                      |
| ervations |                                                          |
| on the    | **Approach (A)**                                         |
| In        |                                                          |
| dependent | > **Definition**: "Approach" refers to the methods used  |
| Review    | > by an organization to address the Baldrige Criteria    |
| Wor       | > Item requirements in Categories 1--6. Approach         |
| ksheet.** | > includes the appropriateness of the methods to the     |
|           | > Item requirements.                                     |
| ![](me    |                                                          |
| dia/image | -   Is the approach systematic (i.e., with repeatable    |
| 6.jpeg){w |     steps, inputs, outputs, time frames)?                |
| idth="1.0 |                                                          |
| 868055555 | -   Is there evidence that the approach is effective?    |
| 555556in" |                                                          |
| h         | -   Is this approach (or collection of approaches) a key |
| eight="0. |     organizational process? Is the approach important to |
| 93125in"} |     the applicant's overall performance?                 |
|           |                                                          |
|           | **Deployment (D)**                                       |
|           |                                                          |
|           | > **Definition**: **"**Deployment" refers to the extent  |
|           | > to which an approach is applied in addressing the      |
|           | > requirements of a Baldrige Criteria Item. Deployment   |
|           | > is evaluated on the basis of the breadth and depth of  |
|           | > the application of the approach to relevant work units |
|           | > throughout the organization.                           |
|           |                                                          |
|           | -   Is deployment addressed?                             |
|           |                                                          |
|           | -   What evidence is presented that the approach is in   |
|           |     use in one, some, or all appropriate work units,     |
|           |     facilities, locations, shifts, organizational        |
|           |     levels, and so forth?                                |
+-----------+----------------------------------------------------------+
| **Analyze | **Learning (L)**                                         |
| the       |                                                          |
| ap        | > **Definition: "**Learning," in the context of the      |
| plicant's | > evaluation factors, refers to new knowledge or skills  |
| response  | > acquired through evaluation, study, experience, and    |
| to each   | > innovation.                                            |
| Process   |                                                          |
| Item, and | -   Has the approach been evaluated and improved? If it  |
| record    |     has, was the evaluation and improvement conducted in |
| your      |     a fact-based, systematic manner (e.g., was it        |
| obs       |     regular, recurring, data driven)?                    |
| ervations |                                                          |
| on the    | -   Is there evidence of organizational learning (i.e.,  |
| In        |     evidence that the learning from this approach is     |
| dependent |     shared with other organizational units/other work    |
| Review    |     processes)?                                          |
| Worksheet |                                                          |
| (cont     | -   Is there evidence of innovation and refinement from  |
| inued).** |     organizational analysis and sharing (e.g., evidence  |
|           |     that the learning is actually used to drive          |
| ![](me    |     innovation and refinement)?                          |
| dia/image |                                                          |
| 6.jpeg){w | **Integration (I)**                                      |
| idth="1.0 |                                                          |
| 868055555 | > **Definition:** As a process evaluation factor,        |
| 555556in" | > "integration" covers the range from organizational     |
| h         | > "alignment" of approaches in the lower scoring ranges  |
| eight="0. | > to "integration" of approaches in the higher ranges.   |
| 93125in"} | >                                                        |
|           | > "Alignment" refers to the consistency of plans,        |
|           | > processes, information, resource decisions, actions,   |
|           | > results, and analyses to support key organization-wide |
|           | > goals. It requires the use of complementary measures   |
|           | > and information for planning, tracking, analysis, and  |
|           | > improvement at three levels: the organization level,   |
|           | > the key process level, and the work unit level.        |
|           | >                                                        |
|           | > "Integration" refers to the harmonization of plans,    |
|           | > processes, information, resource decisions, actions,   |
|           | > results, and analyses to support key organization-wide |
|           | > goals. Effective integration goes beyond alignment and |
|           | > is achieved when the individual components of a        |
|           | > performance management system operate as a fully       |
|           | > interconnected unit.                                   |
|           |                                                          |
|           | -   How well is the approach aligned with the            |
|           |     organizational needs the applicant has identified in |
|           |     other Criteria Items and the Organizational Profile? |
|           |                                                          |
|           | -   Does the applicant indicate complementary measures   |
|           |     and information used for planning, tracking,         |
|           |     analysis, and improvement on three levels: the       |
|           |     organizational level, the key process level, and the |
|           |     department or work-unit level?                       |
|           |                                                          |
|           | -   How well is the approach integrated with             |
|           |     organizational needs?                                |
|           |                                                          |
|           | > *Examples of needs are generally listed as key         |
|           | > factors---strategic challenges, objectives, and        |
|           | > related action plans; organizational mission, vision,  |
|           | > and goals; key processes and measures; key             |
|           | > customer/market segments and requirements; and         |
|           | > employee groups and requirements*.                     |
+-----------+----------------------------------------------------------+
| **        | -   Review your observations.                            |
| Determine |                                                          |
| high      | -   Bold the text of high-priority observations---those  |
| -priority |     that are most significant to your evaluation of the  |
| obse      |     applicant---whether these highlight positive         |
| rvations, |     attributes or areas that are lacking. (Bolding these |
| and bold  |     observations will help your teammates understand     |
| the text  |     what you thought was most important as they          |
| on the    |     synthesize comments during Consensus Review.)        |
| In        |                                                          |
| dependent |                                                          |
| Review    |                                                          |
| Wor       |                                                          |
| ksheet.** |                                                          |
+-----------+----------------------------------------------------------+
| **        | -   In the middle section of the worksheet, note any key |
| Determine |     Criteria requirements or relevant key factors the    |
| if any    |     applicant has not addressed. ***These are known as   |
| important |     "gaps" and should be noted.***                       |
| Criteria  |                                                          |
| req       | > *Remember, not all requirements are of equal           |
| uirements | > importance to all applicants. The intent is to         |
| or        | > identify gaps in the response that are significant for |
| relevant  | > the applicant because they are related to a critical   |
| key       | > key factor.*                                           |
| factors   |                                                          |
| are not   | -   If appropriate, note in the "Overall Item Notes"     |
| a         |     section of the worksheet any critical thoughts that  |
| ddressed, |     relate to the assessment of the Item (i.e., issues   |
| and       |     that cut across several processes within the Item    |
| complete  |     and that you do not want to forget). Do not record   |
| the lower |     Item-level or potential key theme comments here; you |
| section   |     will draft comments during Consensus Review.         |
| of the    |                                                          |
| wor       |                                                          |
| ksheet.** |                                                          |
+-----------+----------------------------------------------------------+
| **Score   | -   Use the Independent Review Worksheet and the scoring |
| the Item  |     grid to determine a scoring range for each process   |
| using the |     evaluation factor.                                   |
| a         |                                                          |
| ssociated | 1.  Review the Criteria requirements.                    |
| scoring   |                                                          |
| grid.**   | 2.  On the Independent Review Worksheet, begin with the  |
|           |     "approach" evaluation factor column. Review your     |
| ![](      |     observations for the Item, particularly noting the   |
| media/ima |     bolded observations.                                 |
| ge7.jpeg) |                                                          |
| {width="1 | 3.  Note the balance and importance of these             |
| .09375in" |     observations relative to the Item requirements and   |
| h         |     key factors.                                         |
| eight="0. |                                                          |
| 94375in"} | 4.  Using the first row of the scoring grid on the next  |
|           |     page (labeled "Approach"), identify which scoring    |
|           |     range of the six provided is most descriptive of the |
|           |     organization's achievement level. Once you have      |
|           |     selected a range, read the descriptions to the left  |
|           |     and to the right of it to ensure that you have       |
|           |     selected the most appropriate one. Place an "x" in   |
|           |     the space below the description of the range you     |
|           |     selected.                                            |
|           |                                                          |
|           | 5.  Go back to the Independent Review Worksheet and      |
|           |     follow steps 1--4 above for each of the remaining    |
|           |     evaluation factors: deployment, learning, and        |
|           |     integration.                                         |
|           |                                                          |
|           | -   Determine the applicant's overall scoring range for  |
|           |     the Item. This range takes into account the          |
|           |     descriptions of all the individual evaluation        |
|           |     factors. Considering the applicant's performance     |
|           |     holistically, place an "x" in the box next to one of |
|           |     the ranges listed in the lower left portion of the   |
|           |     Scoring page.                                        |
|           |                                                          |
|           | > *The applicant does not need to demonstrate all the    |
|           | > characteristics in the selected range; rather, the     |
|           | > score is based on a holistic view of the four          |
|           | > evaluation factors in aggregate.*                      |
|           |                                                          |
|           | -   As a check, read the range descriptions to the left  |
|           |     and the right of the selected range to determine     |
|           |     where the applicant's score falls within that range. |
|           |                                                          |
|           | -   Finally, determine a percentage score that is a      |
|           |     multiple of 5 for the Item. Record the percentage    |
|           |     score in the space provided at the lower right side  |
|           |     of the Scoring page.                                 |
+-----------+----------------------------------------------------------+
| **Read    | > RESULTS ITEM EVALUATION STEPS                          |
| and begin | >                                                        |
| to        | > For each Item, complete the six-step Item Evaluation   |
| evaluate  | > Process, and record observations on the Independent    |
| each      | > Review Worksheet.                                      |
| Results   |                                                          |
| Item in   | 1.  **Read the specific Criteria Results Item to refresh |
| Category  |     your memory and understand the Item requirements.**  |
| 7.**      |                                                          |
|           | 2.  **Determine the most relevant key factors for each   |
| ![](m     |     Item, and record them on the appropriate Independent |
| edia/imag |     Review Worksheet.**                                  |
| e1.jpeg){ |                                                          |
| width="1. |     -   Review your Key Factors Worksheet to determine   |
| 101388888 |         the attributes of the organization that would    |
| 888889in" |         influence its responses to the Item              |
| height="0 |         requirements.                                    |
| .9375in"} |                                                          |
|           |     -   Select the most relevant four to six key factors |
| ![](m     |         for the Item. This will be a subset of those     |
| edia/imag |         listed on the Key Factors Worksheet.             |
| e2.jpeg){ |                                                          |
| width="1. |     -   Record these four to six key factors and their   |
| 101388888 |         corresponding numbers in the space provided at   |
| 888889in" |         the top of the Independent Review Worksheet.     |
| height="0 |                                                          |
| .9375in"} | 3.  **Read the relevant section of the application.**    |
|           |                                                          |
| ![](m     | 4.  **Highlight, flag, mark up, and/or take notes as     |
| edia/imag |     needed.**                                            |
| e3.jpeg){ |                                                          |
| width="1. | 5.  **Refer to the measures or indicators for which you  |
| 101388888 |     expected to see results.**                           |
| 888889in" |                                                          |
| h         |     -   Note any ideas, threads, or patterns that recur  |
| eight="0. |         in multiple Items or Categories. These may be    |
| 93125in"} |         added to your initial set of key themes.         |
|           |                                                          |
| ![](me    | 6.  **Continue to develop the Independent Review         |
| dia/image |     Worksheets.**                                        |
| 8.jpeg){w |                                                          |
| idth="1.0 |     -   Review all figures individually, determine the   |
| 479166666 |         applicant's overall results, and group them as   |
| 666666in" |         appropriate in your observations.                |
| hei       |                                                          |
| ght="0.88 |     -   In the first column, note the specific Area      |
| 958333333 |         addressed (e.g., 7.4\[a\]1).                     |
| 33333in"} |                                                          |
|           |     -   Identify and note in the second column the       |
|           |         reported results that you are evaluating (e.g.,  |
|           |         Figures 7.4-1, 7.4-2).                           |
|           |                                                          |
|           |     -   Identify and note in the third column the        |
|           |         numbers of the relevant key factors that         |
|           |         influence this result (e.g., employee segment,   |
|           |         shift, location, type).                          |
|           |                                                          |
|           | > *Refer to the notes you took for each Process Item on  |
|           | > results that you expected to see. You do not need to   |
|           | > provide observations on every figure in the            |
|           | > application. Instead, focus on figures that are        |
|           | > responsive to the Criteria requirements and address    |
|           | > important customer, product and service, market,       |
|           | > process, and action-plan performance requirements      |
|           | > identified in the Organizational Profile and in        |
|           | > Process Items, or identify the results that are        |
|           | > missing in these important areas. **Mark favorable     |
|           | > observations with a "+" sign and unfavorable           |
|           | > observations with a "-" sign.***                       |
+-----------+----------------------------------------------------------+
| **Analyze | -   **As you study the reported results, keep the        |
| the       |     following factors and questions in mind.**           |
| results   |                                                          |
| to        | **Performance Levels (Le)**                              |
| determine |                                                          |
| how well  | **Definition**: "Performance levels" refer to numerical  |
| the       | information that places or positions an organization's   |
| applicant | results and performance on a meaningful measurement      |
| is        | scale. Performance levels permit evaluation relative to  |
| perf      | past performance, projections, goals, and appropriate    |
| orming.** | comparisons.                                             |
|           |                                                          |
| ![](me    | -   What levels are provided?                            |
| dia/image |                                                          |
| 6.jpeg){w | -   Is the measurement scale meaningful?                 |
| idth="1.0 |                                                          |
| 868055555 | -   Are key results missing?                             |
| 555556in" |                                                          |
| h         | **Trends (T)**                                           |
| eight="0. |                                                          |
| 93125in"} | > **Definition**: "Trends" refer to numerical            |
|           | > information that shows the direction and rate of       |
|           | > change for an organization's results. A minimum of     |
|           | > three data points generally is needed to begin to      |
|           | > ascertain a trend.                                     |
|           |                                                          |
|           | -   Are trends provided for few, many, or most Areas     |
|           |     addressed in the Item requirements?                  |
|           |                                                          |
|           | -   Is the interval between measures or frequencies      |
|           |     appropriate?                                         |
|           |                                                          |
|           | -   Are the trends positive, negative, or flat?          |
|           |                                                          |
|           | -   What is the rate of change (slope of the trend)?     |
|           |                                                          |
|           | -   Do the trends demonstrate little, some, or much      |
|           |     breadth in the applicant's improvement efforts       |
|           |     (i.e., how widely are they deployed and shared)?     |
|           |                                                          |
|           | -   Are significant variations in trends explained in    |
|           |     the text of the application?                         |
+-----------+----------------------------------------------------------+
| **Analyze | **Comparisons (C)**                                      |
| the       |                                                          |
| results   | > **Definition**: **"**Comparisons" refer to how the     |
| to        | > applicant's results compare with the results of other  |
| determine | > organizations. Comparisons can be made to the results  |
| how well  | > of competitors, organizations providing similar        |
| the       | > products and services, industry averages, or           |
| applicant | > best-in-class organizations. The maturity of the       |
| is        | > organization should help determine what comparisons    |
| p         | > are most relevant.                                     |
| erforming |                                                          |
| (cont     | -   Are comparisons provided?                            |
| inued).** |                                                          |
|           | -   Are the comparisons to key competitors, industry     |
| ![](me    |     sector averages, or best-in-class organizations?     |
| dia/image |                                                          |
| 6.jpeg){w | -   How does the applicant compare against these other   |
| idth="1.0 |     organizations?                                       |
| 868055555 |                                                          |
| 555556in" | **Integration (I)**                                      |
| h         |                                                          |
| eight="0. | > **Definition**: **"**Integration" refers to the extent |
| 93125in"} | > to which results measures (often through segmentation) |
|           | > address important customer, product and service,       |
|           | > market, process, and action-plan performance           |
|           | > requirements identified in the Organizational Profile  |
|           | > and in Process Items; include valid indicators of      |
|           | > future performance; and are harmonized across          |
|           | > processes and work units to support organization-wide  |
|           | > goals.                                                 |
|           |                                                          |
|           | -   To what extent do results link to key factors and    |
|           |     Process Items?                                       |
|           |                                                          |
|           | -   Are results segmented appropriately (e.g., by key    |
|           |     customer, patient, or student segment; employee      |
|           |     type; process/education program or service; or       |
|           |     geographic location) to help the applicant improve?  |
+-----------+----------------------------------------------------------+
| **        | -   Review your observations.                            |
| Determine |                                                          |
| high      | -   Make sure you have noted any missing results in      |
| -priority |     areas of importance.                                 |
| obse      |                                                          |
| rvations, | -   Determine and bold the text of those observations    |
| and bold  |     that are most significant to your evaluation of the  |
| the text  |     applicant, whether these indicate positive,          |
| on the    |     negative, or missing results.                        |
| wor       |                                                          |
| ksheet.** |                                                          |
+-----------+----------------------------------------------------------+
| **        | -   Note in the middle section of the worksheet any key  |
| Determine |     Results requirements or relevant key factors that    |
| if any    |     are not addressed. ***These are known as "gaps" and  |
| important |     should be noted.***                                  |
| Criteria  |                                                          |
| req       | > *Your aim is to identify gaps in the response that are |
| uirements | > significant for this applicant because they are        |
| or        | > related to a critical key factor.*                     |
| relevant  |                                                          |
| key       | -   If appropriate, note in the "Overall Item Notes"     |
| factors   |     section any critical thoughts that relate to the     |
| are not   |     assessment of the Item (e.g., positive trends, lack  |
| a         |     of segmentation for a specific group). Do not record |
| ddressed, |     Item-level or potential key theme comments here; you |
| and       |     will draft comments during the Consensus Review.     |
| complete  |                                                          |
| the lower |                                                          |
| section   |                                                          |
| of the    |                                                          |
| wor       |                                                          |
| ksheet.** |                                                          |
+-----------+----------------------------------------------------------+
| **Score   | -   Use the Independent Review Worksheet and the scoring |
| the Item  |     grid to determine a scoring range for each results   |
| using the |     evaluation factor. []{.mark}                         |
| a         |                                                          |
| ssociated | 1.  Review the Criteria requirements.                    |
| scoring   |                                                          |
| grid.**   | 2.  On the worksheet, review the column labeled          |
|           |     "Levels." Particularly note the bolded observations. |
| ![](      |                                                          |
| media/ima | 3.  Note the balance and importance of these             |
| ge7.jpeg) |     observations relative to the Item requirements and   |
| {width="1 |     key factors.                                         |
| .09375in" |                                                          |
| h         | 4.  Using the first row of the scoring grid, labeled     |
| eight="0. |     "Levels," identify which scoring range of the six    |
| 94375in"} |     provided is most descriptive of the organization's   |
|           |     achievement level. Once you have selected a range,   |
|           |     read the descriptions to the left and to the right   |
|           |     of it to ensure that you have selected the most      |
|           |     appropriate one. Place an "x" in the space below the |
|           |     description of the range you select.                 |
|           |                                                          |
|           | 5.  Go back to the Independent Review Worksheet, and     |
|           |     follow steps 1--4 for each of the remaining          |
|           |     evaluation factors---trends, comparisons, and        |
|           |     integration---particularly noting missing results or |
|           |     "gaps."                                              |
|           |                                                          |
|           | -   Determine the applicant's overall range for the      |
|           |     Item. This range takes into account the descriptions |
|           |     of all the individual evaluation factors.            |
|           |     Considering the applicant's performance              |
|           |     holistically, place an "x" in the box next to the    |
|           |     appropriate range listed in the lower left portion   |
|           |     of the Scoring page.                                 |
|           |                                                          |
|           | > *The applicant does not need to demonstrate all the    |
|           | > characteristics in the selected range; rather, the     |
|           | > score is based on a holistic view of the four          |
|           | > evaluation factors in aggregate.*                      |
|           |                                                          |
|           | -   As a check, read the range descriptions to the left  |
|           |     and the right of the selected range to determine     |
|           |     where the applicant's score falls within that range. |
|           |                                                          |
|           | -   Finally, determine a percentage score that is a      |
|           |     multiple of 5 for the Item. Record the percentage    |
|           |     score in the space provided at the lower right side  |
|           |     of the Scoring page.                                 |
+-----------+----------------------------------------------------------+
| **Review  | -   If you drafted an initial set of responses to the    |
| all       |     questions on the Key Themes Worksheet, review and    |
| In        |     update the responses to reflect your entire set of   |
| dependent |     observations.                                        |
| Review    |                                                          |
| Work      |                                                          |
| sheets.** |                                                          |
+-----------+----------------------------------------------------------+
| **Record  | -   Enter all Item scores into the Score Summary         |
| all       |     Worksheet near the end of the scorebook.             |
| scores on |                                                          |
| the Score | > *The Score Summary Worksheet is programmed to          |
| Summary   | > calculate the point scores for you.*                   |
| Wor       |                                                          |
| ksheet.** | -   Review and reconcile your analysis of the            |
|           |     application with the Grand Total (line D on the      |
|           |     Score Summary Worksheet).                            |
+-----------+----------------------------------------------------------+
| *         | -   Compare the key factors on your Independent Review   |
| *Finalize |     Worksheets with the initial list on your Key Factors |
| the key   |     Worksheet.                                           |
| f         |                                                          |
| actors.** | -   Add additional key factors identified through the    |
|           |     evaluation process.                                  |
|           |                                                          |
|           | -   Determine whether to add any of your originally      |
|           |     identified but unused key factors to your            |
|           |     Independent Review Worksheets. Alternatively,        |
|           |     consider eliminating any unused key factors from the |
|           |     Key Factors Worksheet.                               |
+-----------+----------------------------------------------------------+
| **(       | -   Review all observations---in particular bolded       |
| Required) |     ones---to identify ideas, threads, and patterns that |
| Finalize  |     recur.                                               |
| responses |                                                          |
| to the    | -   Refer to the Key Themes Worksheet, which contains a  |
| questions |     set of questions. You are not required to respond to |
| under a,  |     all the questions. Instead, use them to prompt your  |
| b, c, and |     thinking. The balance of responses across the        |
| d on the  |     questions should reflect the overall balance of      |
| Key       |     observations across the Item (e.g., for a            |
| Themes    |     high-scoring applicant, you would note more "a"      |
| Wor       |     strengths than "b" opportunities and more "c"        |
| ksheet.** |     strengths than "d" opportunities).                   |
+-----------+----------------------------------------------------------+
| **Record  | -   Record the information requested on the front cover  |
| the       |     of the scorebook, including the time spent           |
| in        |     completing it.                                       |
| formation |                                                          |
| requested |                                                          |
| on the    |                                                          |
| front     |                                                          |
| cover.**  |                                                          |
+-----------+----------------------------------------------------------+
| *         | -   Cover sheet                                          |
| *Assemble |                                                          |
| the       | -   Completed Conflict of Interest Determination         |
| scorebook |     Worksheet                                            |
| using the |                                                          |
| p         | -   Signed Conflict of Interest Statement                |
| rescribed |                                                          |
| se        | -   Key Factors Worksheet                                |
| quence.** |                                                          |
|           | -   Key Themes Worksheet                                 |
|           |                                                          |
|           | -   One Independent Review Worksheet for each Item in    |
|           |     ascending numerical order (e.g., 1.1, 1.2)           |
|           |                                                          |
|           | -   Score Summary Worksheet                              |
+-----------+----------------------------------------------------------+
| **Upload  | -   **Upload your scorebook to the secure Web site,      |
| to        |     *examinerdepot.***                                   |
| *examiner |                                                          |
| depot.*** |                                                          |
+-----------+----------------------------------------------------------+

#