Baldrige National Quality Program • National Institute of Standards and
Technology • Department of Commerce

> **Consensus Scorebook**
>
> 2008

  ------------- ------- -------------------- ---------------- ------- -------- -------- ------- ------- ---------------- --
  Examiner's                                                                   Number                                    
  Name                                                                         of Hours                                  
                                                                               Worked                                    

  Applicant                                                                                                              
  Number                                                                                                                 

                                                                                                                         

                                                                                                                         

  **Criteria,                                                                                                            
  Score Summary                                                                                                          
  Worksheet,                                                                                                             
  and Scoring                                                                                                            
  Guidelines                                                                                                             
  Used:**                                                                                                                

                        Business/Nonprofit                            Health                            Education        
                                                                      Care                                               
  ------------- ------- -------------------- ---------------- ------- -------- -------- ------- ------- ---------------- --

Item Worksheet---Item 3.1

+-----------------------------------------------------------------------+
| **IMPORTANT! *Save this document using one of the following file      |
| names (as appropriate):***                                            |
|                                                                       |
| **[Your Role File Name]{.underline}**                                 |
|                                                                       |
| Item lead, posting version 1 Item 3_1v1                               |
|                                                                       |
| Item backup, posting changes to version 1 Item 3_1BU                  |
|                                                                       |
| Item lead, posting version 2 Item 3_1v2                               |
|                                                                       |
| Team member, posting feedback Item 3_1v2\_\[insert your initials\]    |
|                                                                       |
| Item lead, posting consensus-call-ready version Item 3_1v3            |
|                                                                       |
| Item lead, posting Consensus Scorebook version Item 3_1_consensus     |
+-----------------------------------------------------------------------+

> **Rationale for Item 3.1 Version v1**

**Instructions:** Summarize your rationale for selecting and
synthesizing the issues identified in your proposed comments (e.g.,
linkage to a critical key factor; a possible role-model practice or
result; an issue that may be preventing the applicant from scoring in a
higher range; a general deployment theme across most Independent Review
Worksheets; team members LJ, HH, DB, PS included this issue) and the
scoring range and score.

Note: This tool may be useful for summarizing your analysis before you
draft comments.

+-----+----------------------------------------------------------------+
| **S | **Summary of Proposed Strengths**                              |
| tre |                                                                |
| ngt | (Add or delete rows as needed. Aim for around 6 total combined |
| h** | strengths and opportunities for improvement \[OFIs\].)         |
|     |                                                                |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   | a(1) Seven key niche customer groups. New market segments      |
|     | identified by Marketing & Public Relations Team during SPP.    |
|     | Data and information fed to Product Engineering and Design     |
|     | Team for product development.(nj, mb, sk, rn) *integrated with |
|     | SPP*                                                           |
+-----+----------------------------------------------------------------+
| 2   | a(2) Five VOC Processes (Fig. 3.1-2): customer complaints,     |
|     | market research, customer surveys, customer relationship       |
|     | management, and customer advisory group (rh, nj, rn, mb, sk,   |
|     | jj) VOC approaches vary in their methods, locations, and       |
|     | frequency of use. (rh) *VOC methods touch all customer         |
|     | segments*                                                      |
+-----+----------------------------------------------------------------+
| 3   | a(3) Data available to all employees through MAP to use in     |
|     | process improvements and sent to retailers and carrier on a    |
|     | quarterly basis(nj, sk, jj, rh); data and information analyzed |
|     | and used as inputs to the SPP and to the Product, Feature, and |
|     | Process Development (PFPD) Process (sk, mb, rh). *Well         |
|     | deployed to all employees & partners*                          |
+-----+----------------------------------------------------------------+
| 4   | a(4) all VOC processes reviewed annually during the Process    |
|     | Improvement Process (PIP) before the SPP (nj, mb, jj, rn);     |
|     | findings from Allegiance Survey reviewed by process owners &   |
|     | internal stakeholders at annual Improvement Day (nj, mb, jj,   |
|     | rh, sk)                                                        |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| *   | **Summary of Proposed Opportunities for Improvement**          |
| *OF |                                                                |
| I** | (Add or delete rows as needed. Aim for around 6 total combined |
|     | strengths and OFIs.)                                           |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   | a(1) Not clear how applicant includes customers of competitors |
|     | in determination of market segments to pursue. (rh, rn, sk)    |
+-----+----------------------------------------------------------------+
| 2   | a(2) **Systematic process not described for how VOC data is    |
|     | used to determine customer requirements, needs, and            |
|     | expectations.** How applicant determines needed improvements   |
|     | in its work system and processes. (**mb,** nj) *Many* *of us   |
|     | missed these, but they are key requirements and a big          |
|     | vulnerability. Recommend bolding this OFI.*                    |
+-----+----------------------------------------------------------------+
| 3   | a(1, 2) **Not clear how applicant uses information from former |
|     | customers or how listening methods vary for different customer |
|     | groups, market segments**. (**mb**, **nj**, rn, **rh**, sk) *I |
|     | agree with others on the team re: bolding this OFI -- failing  |
|     | to vary VOC by customer groups/market segments is a big gap    |
|     | for this applicant.*                                           |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
|     | **Summary of KEY Observations Not Used**                       |
|     |                                                                |
|     | (Add or delete rows as needed.)                                |
+-----+----------------------------------------------------------------+
| 1   | a(1) requirements of retailers and carrier not addressed (jj,  |
|     | rh) *This is prescriptive -- applicant describes retailers and |
|     | carrier as partners, not customers.*                           |
+-----+----------------------------------------------------------------+
| 2   | a(2) effective, systematic approach to determining key         |
|     | customer requirements. (rh, jj, rn, sk)                        |
|     |                                                                |
|     | *This conflicts with OFI #2 -- applicant does not describe how |
|     | customer requirements are determined.*                         |
+-----+----------------------------------------------------------------+
| 2   | a(4) **keeping listening and learning methods current not well |
|     | addressed** (**rn**) *Conflicts with Strength #4*              |
+-----+----------------------------------------------------------------+

**Proposed Scoring Range and Score**

  ------------- ------------ ----------------------------------------------------
  **Process**   **Range      **Summarize below why this range is most
                selected**   appropriate.**

  Approach      30-45        Since the applicant didn't respond to the 3.1a(2)
                             requirement -- "how do you use the VOC to determine
                             key customer requirements," I can't justify scoring
                             any higher. They did not respond to an important
                             part of the overall requirements.

  Deployment    50-65        VOC methods are well deployed to all customer
                             segments except customers of competitors, former
                             customers, and employees who are also considered
                             customers. In addition, the applicant does not vary
                             their VOC for these different segments. VOC data is
                             deployed to all employees and partners, however. I
                             think the gaps with the segments keep them out of
                             the 70-85% range.

  Learning      30-45        Beginning of systematic approach to improvement --
                             applicant tells us they review things to make
                             improvements, but provides little evidence of
                             improvements they've actually made.

  Integration   70-85        Approaches are well integrated with SPP, MAP
                             database, and PFPD
  ------------- ------------ ----------------------------------------------------

**For the Item as a whole, the range most representative of the
organization's achievement level is [50-65%]{.underline}, with a score
of [50]{.underline}.**

Additional scoring justification not included above:

*Not sure if the 2 bolded OFIs should result in a lower scoring range of
30-45. I'm looking for input from the team.*

**Item Worksheet---Item 3.1**

Indicate the 4--6 most important key business/organization factors
relevant to this Item.

  -----------------------------------------------------------------------
  Five major product lines: Novel Complete, Novel Secure 1, Novel Free,
  Novel Bug, and NovelAid

  Key customer segments: personal use consumers (students in Gen-Y,
  celebrities and sports stars, preteens, single adult females, elderly,
  disabled); personal/business consumers ("outdoors people"); business
  use consumers (truckers, taxi drivers); government (emergency services
  workers, the Department of Homeland Security)

  Key customer requirements: All - ease of use, reliability; Personal
  consumers -- Trendiness, convenience, secure/encrypted data and
  transmission, personal/home safety and security, low cost, ruggedness;
  Business consumers -- Ruggedness, personal safety and security, data
  and voice capability, sustained signal/strength across distances,
  secure/encrypted data and transmission; Government consumers --
  Security, data and voice capability, secure/encrypted data and
  transmission, sustained signal/strength across distances

  Success factors: Relationship with Carriers, Time to market with new
  products, hardware/software quality, process performance and its
  positive impact on margins, supply chain management, and collaborations
  with key suppliers/partners

  Strategic Challenges: availability of highly skilled workforce,
  communication, logistics, rapidly changing customer/market needs,
  protection of intellectual property, volatility of overseas
  environment, market forces driving the cost of cell phone and market
  penetration

  Strategic Advantages: product/feature design innovation, business model
  innovation, lowered costs from offshore supplier/partnership
  relationships
  -----------------------------------------------------------------------

**Strengths and Opportunities for Improvement**

Indicate the relative importance/strength of the comment by bolding the
text for ++ or - - comments.

**STRENGTHS**

(Tab to move to the next column; tab from the final column to begin the
next comment.)

  -------------------------------------------------------------------------------------------
  **Item Ref.**               **STRENGTHS**
  --------------------------- ---------------------------------------------------------------
  3.1a(1)                     During the SPP, the Marketing and Public Relations Team reviews
                              existing market segments and identifies new segments. The team
                              has identified seven key niche customer groups (Figure 3.1-1)
                              that are subsegments of the three main market segments
                              (personal, business, and government consumers). Market data and
                              information are fed to the organization's Product Engineering
                              and Design Team to develop new products/features. A cycle of
                              improvement resulted in reversing the sequence of incorporating
                              customer input into the process; information from customers and
                              potential customers is now considered before identifying
                              potential products and product features.

  3.1a(2)                     To gather information on customer requirements, needs, and
                              changing expectations, the applicant has established Five
                              Voices of the Customer (VOC, Figure 3.1-2): customer
                              complaints, market research, customer surveys, customer
                              relationship management, and customer advisory group
                              communications. These mechanisms vary in their methods,
                              locations, and frequency of use.

  3.1a(3)                     VOC data are available to all employees in the MAP database for
                              use in process improvements and are fed to the Customer Service
                              Team, which oversees the call center. VOC data are sent to
                              retailers and the carrier on a quarterly basis. All customer
                              data and information are analyzed and used as inputs to the SPP
                              and to the Product, Feature, and Process Development (PFPD)
                              Process.

  *OPPORTUNITIES FOR          **To help keep its VOC methods current, the Marketing and
  IMPROVEMENT                 Public Relations Team reviews them annually during the Process
  **3.1a(1,2)*****3.1a(4)**   Improvement Process (PIP) that precedes the SPP. In addition,
                              the Allegiance Survey, which is reviewed at the annual
                              Improvement Day, includes a question about ways to improve
                              listening and learning methods.**
  -------------------------------------------------------------------------------------------

While plans are in place to seek information from customers of
competitors, it is unclear how the applicant currently includes
customers of competitors and other potential customers and markets in
its determination of customer groups and market segments. Without
considering the requirements and expectations of all customer groups,
the applicant may not succeed in fulfilling the requirements of all of
its target customers and markets and may miss opportunities to penetrate
new niche markets with innovative products.3.1a(2)**While the applicant
uses the Five Voices of the Customer (Figure 3.1-2) to gather
information from and about customers, a systematic process is not
described for using this information to determine customer requirements,
needs, and expectations or to determine needed improvements in its work
system and processes. Without such a systematic process, the
organization may not be able to fully understand its customers'
purchasing decisions, achieve customer loyalty, or build customer
relationships.** 3.1a(2)**It is not clear how the organization's
listening and learning methods include former customers or vary for its
customers, customer groups, and market segments. The absence of a
systematic process to tailor its listening and learning methods for its
diverse customers (which range from preteens to the Department of
Homeland Security) or to most effectively use the information gathered
from its numerous sources may make it difficult for the organization to
address customer requirements and achieve its vision to be the most
innovative company for mobile communication in the world.**

**Evaluation Factor Score Summary---Item 3.1**

  ------------- ---------------- -------------- ---------------- ---------------- ---------------------- -------------------
  **Factor**    **0--5%**        **10--25%**    **30--45%**      **50--65%**      **70--85%**            **90--100%**

  Approach      No systematic    The beginning  An effective,    An effective,    An effective,          An effective,
                approach to Item of a           systematic       systematic       systematic approach,   systematic
                requirements is  systematic     approach,        approach,        responsive to the      approach, fully
                evident;         approach to    responsive to    responsive to    multiple requirements  responsive to the
                information is   the basic      the basic        the overall      of the Item, is        multiple
                anecdotal.       requirements   requirements of  requirements of  evident.               requirements of the
                                 of the Item is the Item, is     the Item, is                            Item, is evident.
                                 evident.       evident.         evident.                                

                                                X                                                        

  Deployment    Little or no     The approach   The approach is  The approach is  The approach is well   The approach is
                deployment of    is in the      deployed,        well deployed,   deployed, with no      fully deployed
                any systematic   early stages   although some    although         significant gaps.      without significant
                approach is      of deployment  areas or work    deployment may                          weaknesses or gaps
                evident.         in most areas  units are in     vary in some                            in any areas or
                                 or work units, early stages of  areas or work                           work units.
                                 inhibiting     deployment.      units.                                  
                                 progress in                                                             
                                 achieving the                                                           
                                 basic                                                                   
                                 requirements                                                            
                                 of the Item.                                                            

                                                                 X                                       

  Learning      An improvement   Early stages   The beginning of A fact-based,    Fact-based, systematic Fact-based,
                orientation is   of a           a systematic     systematic       evaluation and         systematic
                not evident;     transition     approach to      evaluation and   improvement and        evaluation and
                improvement is   from reacting  evaluation and   improvement      organizational         improvement and
                achieved through to problems to improvement of   process and some learning, including    organizational
                reacting to      a general      key processes is organizational   innovation, are key    learning through
                problems.        improvement    evident.         learning,        management tools;      innovation are key
                                 orientation                     including        there is clear         organization-wide
                                 are evident.                    innovation, are  evidence of refinement tools; refinement
                                                                 in place for     as a result of         and innovation,
                                                                 improving the    organizational-level   backed by analysis
                                                                 efficiency and   analysis and sharing.  and sharing, are
                                                                 effectiveness of                        evident throughout
                                                                 key processes.                          the organization.

                                                X                                                        

  Integration   No               The approach   The approach is  The approach is  The approach is        The approach is
                organizational   is aligned     in the early     aligned with     integrated with        well integrated
                alignment is     with other     stages of        organizational   organizational needs   with organizational
                evident;         areas or work  alignment with   needs identified identified in response needs identified in
                individual areas units largely  basic            in response to   to the Organizational  response to the
                or work units    through joint  organizational   the              Profile and other      Organizational
                operate          problem        needs identified Organizational   Process Items.         Profile and other
                independently.   solving.       in response to   Profile and                             Process Items.
                                                the              other Process                           
                                                Organizational   Items.                                  
                                                Profile and                                              
                                                other Process                                            
                                                Items.                                                   

                                                                                  X                      
  ------------- ---------------- -------------- ---------------- ---------------- ---------------------- -------------------

**Guidance:** The overall score is not intended to be a numerical
average of the elements above; the Examiners select the range and score
that are most descriptive of the organization's achievement level for
the Item.

**Item 3.1---Overall Score**

> 0--5%
>
> 10--25%
>
> 30--45%
>
> X 50--65% Item 3.1 Score 50%
>
> 70--85%
>
> 90--100%

Item Worksheet---Item 4.1

+-----------------------------------------------------------------------+
| **IMPORTANT! *Save this document using one of the following file      |
| names (as appropriate):***                                            |
|                                                                       |
| **[Your Role File Name]{.underline}**                                 |
|                                                                       |
| Item lead, posting version 1 Item 4_1v1                               |
|                                                                       |
| Item backup, posting changes to version 1 Item 4_1BU                  |
|                                                                       |
| Item lead, posting version 2 Item 4_1v2                               |
|                                                                       |
| Team member, posting feedback Item 4_1v2\_\[insert your initials\]    |
|                                                                       |
| Item lead, posting consensus-call-ready version Item 4_1v3            |
|                                                                       |
| Item lead, posting Consensus Scorebook version Item 4_1_consensus     |
+-----------------------------------------------------------------------+

> **Rationale for Item 4.1 Version**

**Instructions:** Summarize your rationale for selecting and
synthesizing the issues identified in your proposed comments (e.g.,
linkage to a critical key factor; a possible role-model practice or
result; an issue that may be preventing the applicant from scoring in a
higher range; a general deployment theme across most Independent Review
Worksheets; team members LJ, HH, DB, PS included this issue) and the
scoring range and score.

Note: This tool may be useful for summarizing your analysis before you
draft comments.

+-----+----------------------------------------------------------------+
| **S | **Summary of Proposed Strengths**                              |
| tre |                                                                |
| ngt | (Add or delete rows as needed. Aim for around 6 total combined |
| h** | strengths and opportunities for improvement \[OFIs\].)         |
|     |                                                                |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+
| 3   |                                                                |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| *   | **Summary of Proposed Opportunities for Improvement**          |
| *OF |                                                                |
| I** | (Add or delete rows as needed. Aim for around 6 total combined |
|     | strengths and OFIs.)                                           |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+
| 3   |                                                                |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
|     | **Summary of KEY Observations Not Used**                       |
|     |                                                                |
|     | (Add or delete rows as needed.)                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+

**Proposed Scoring Range and Score**

  ------------- ------------ ----------------------------------------------------
  **Process**   **Range      **Summarize below why this range is most
                selected**   appropriate.**

  Approach                   

  Deployment                 

  Learning                   

  Integration                
  ------------- ------------ ----------------------------------------------------

**For the Item as a whole, the range most representative of the
organization's achievement level is \_\_\_\_, with a score of
\_\_\_\_.**

Additional scoring justification not included above:

**Item Worksheet---Item 4.1**

Indicate the 4--6 most important key business/organization factors
relevant to this Item.

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

**Strengths and Opportunities for Improvement**

Indicate the relative importance/strength of the comment by bolding the
text for ++ or - - comments.

**STRENGTHS**

(Tab to move to the next column; tab from the final column to begin the
next comment.)

  ------------------------------------------------------------------------
  **Item   **STRENGTHS**
  Ref.**   
  -------- ---------------------------------------------------------------
           

  ------------------------------------------------------------------------

**OPPORTUNITIES FOR IMPROVEMENT**

*(Tab to move to the next column; tab from the final column to begin the
next comment.)*

  ------------------------------------------------------------------------
  **Item   **OPPORTUNITIES FOR IMPROVEMENT**
  Ref.**   
  -------- ---------------------------------------------------------------
           

  ------------------------------------------------------------------------

**Evaluation Factor Score Summary---Item 4.1**

  ------------- ---------------- -------------- ---------------- ---------------- ---------------------- -------------------
  **Factor**    **0--5%**        **10--25%**    **30--45%**      **50--65%**      **70--85%**            **90--100%**

  Approach      No systematic    The beginning  An effective,    An effective,    An effective,          An effective,
                approach to Item of a           systematic       systematic       systematic approach,   systematic
                requirements is  systematic     approach,        approach,        responsive to the      approach, fully
                evident;         approach to    responsive to    responsive to    multiple requirements  responsive to the
                information is   the basic      the basic        the overall      of the Item, is        multiple
                anecdotal.       requirements   requirements of  requirements of  evident.               requirements of the
                                 of the Item is the Item, is     the Item, is                            Item, is evident.
                                 evident.       evident.         evident.                                

                                                                                                         

  Deployment    Little or no     The approach   The approach is  The approach is  The approach is well   The approach is
                deployment of    is in the      deployed,        well deployed,   deployed, with no      fully deployed
                any systematic   early stages   although some    although         significant gaps.      without significant
                approach is      of deployment  areas or work    deployment may                          weaknesses or gaps
                evident.         in most areas  units are in     vary in some                            in any areas or
                                 or work units, early stages of  areas or work                           work units.
                                 inhibiting     deployment.      units.                                  
                                 progress in                                                             
                                 achieving the                                                           
                                 basic                                                                   
                                 requirements                                                            
                                 of the Item.                                                            

                                                                                                         

  Learning      An improvement   Early stages   The beginning of A fact-based,    Fact-based, systematic Fact-based,
                orientation is   of a           a systematic     systematic       evaluation and         systematic
                not evident;     transition     approach to      evaluation and   improvement and        evaluation and
                improvement is   from reacting  evaluation and   improvement      organizational         improvement and
                achieved through to problems to improvement of   process and some learning, including    organizational
                reacting to      a general      key processes is organizational   innovation, are key    learning through
                problems.        improvement    evident.         learning,        management tools;      innovation are key
                                 orientation                     including        there is clear         organization-wide
                                 are evident.                    innovation, are  evidence of refinement tools; refinement
                                                                 in place for     as a result of         and innovation,
                                                                 improving the    organizational-level   backed by analysis
                                                                 efficiency and   analysis and sharing.  and sharing, are
                                                                 effectiveness of                        evident throughout
                                                                 key processes.                          the organization.

                                                                                                         

  Integration   No               The approach   The approach is  The approach is  The approach is        The approach is
                organizational   is aligned     in the early     aligned with     integrated with        well integrated
                alignment is     with other     stages of        organizational   organizational needs   with organizational
                evident;         areas or work  alignment with   needs identified identified in response needs identified in
                individual areas units largely  basic            in response to   to the Organizational  response to the
                or work units    through joint  organizational   the              Profile and other      Organizational
                operate          problem        needs identified Organizational   Process Items.         Profile and other
                independently.   solving.       in response to   Profile and                             Process Items.
                                                the              other Process                           
                                                Organizational   Items.                                  
                                                Profile and                                              
                                                other Process                                            
                                                Items.                                                   

                                                                                                         
  ------------- ---------------- -------------- ---------------- ---------------- ---------------------- -------------------

**Guidance:** The overall score is not intended to be a numerical
average of the elements above; the Examiners select the range and score
that are most descriptive of the organization's achievement level for
the Item.

**Item 4.1---Overall Score**

> 0--5%
>
> 10--25%
>
> 30--45%
>
> 50--65% Item 4.1 Score %
>
> 70--85%
>
> 90--100%

Item Worksheet---Item 4.2

+-----------------------------------------------------------------------+
| **IMPORTANT! *Save this document using one of the following file      |
| names (as appropriate):***                                            |
|                                                                       |
| **[Your Role File Name]{.underline}**                                 |
|                                                                       |
| Item lead, posting version 1 Item 4_2v1                               |
|                                                                       |
| Item backup, posting changes to version 1 Item 4_2BU                  |
|                                                                       |
| Item lead, posting version 2 Item 4_2v2                               |
|                                                                       |
| Team member, posting feedback Item 4_2v2\_\[insert your initials\]    |
|                                                                       |
| Item lead, posting consensus-call-ready version Item 4_2v3            |
|                                                                       |
| Item lead, posting Consensus Scorebook version Item 4_2_consensus     |
+-----------------------------------------------------------------------+

> **Rationale for Item 4.2 Version**

**Instructions:** Summarize your rationale for selecting and
synthesizing the issues identified in your proposed comments (e.g.,
linkage to a critical key factor; a possible role-model practice or
result; an issue that may be preventing the applicant from scoring in a
higher range; a general deployment theme across most Independent Review
Worksheets; team members LJ, HH, DB, PS included this issue) and the
scoring range and score.

Note: This tool may be useful for summarizing your analysis before you
draft comments.

+-----+----------------------------------------------------------------+
| **S | **Summary of Proposed Strengths**                              |
| tre |                                                                |
| ngt | (Add or delete rows as needed. Aim for around 6 total combined |
| h** | strengths and opportunities for improvement \[OFIs\].)         |
|     |                                                                |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+
| 3   |                                                                |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| *   | **Summary of Proposed Opportunities for Improvement**          |
| *OF |                                                                |
| I** | (Add or delete rows as needed. Aim for around 6 total combined |
|     | strengths and OFIs.)                                           |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+
| 3   |                                                                |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
|     | **Summary of KEY Observations Not Used**                       |
|     |                                                                |
|     | (Add or delete rows as needed.)                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+

**Proposed Scoring Range and Score**

  ------------- ------------ ----------------------------------------------------
  **Process**   **Range      **Summarize below why this range is most
                selected**   appropriate.**

  Approach                   

  Deployment                 

  Learning                   

  Integration                
  ------------- ------------ ----------------------------------------------------

**For the Item as a whole, the range most representative of the
organization's achievement level is \_\_\_\_, with a score of
\_\_\_\_.**

Additional scoring justification not included above:

**Item Worksheet---Item 4.2**

Indicate the 4--6 most important key business/organization factors
relevant to this Item.

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

**Strengths and Opportunities for Improvement**

Indicate the relative importance/strength of the comment by bolding the
text for ++ or - - comments.

**STRENGTHS**

(Tab to move to the next column; tab from the final column to begin the
next comment.)

  ------------------------------------------------------------------------
  **Item   **STRENGTHS**
  Ref.**   
  -------- ---------------------------------------------------------------
           

  ------------------------------------------------------------------------

**OPPORTUNITIES FOR IMPROVEMENT**

*(Tab to move to the next column; tab from the final column to begin the
next comment.)*

  ------------------------------------------------------------------------
  **Item   **OPPORTUNITIES FOR IMPROVEMENT**
  Ref.**   
  -------- ---------------------------------------------------------------
           

  ------------------------------------------------------------------------

**Evaluation Factor Score Summary---Item 4.2**

  ------------- ---------------- -------------- ---------------- ---------------- ---------------------- -------------------
  **Factor**    **0--5%**        **10--25%**    **30--45%**      **50--65%**      **70--85%**            **90--100%**

  Approach      No systematic    The beginning  An effective,    An effective,    An effective,          An effective,
                approach to Item of a           systematic       systematic       systematic approach,   systematic
                requirements is  systematic     approach,        approach,        responsive to the      approach, fully
                evident;         approach to    responsive to    responsive to    multiple requirements  responsive to the
                information is   the basic      the basic        the overall      of the Item, is        multiple
                anecdotal.       requirements   requirements of  requirements of  evident.               requirements of the
                                 of the Item is the Item, is     the Item, is                            Item, is evident.
                                 evident.       evident.         evident.                                

                                                                                                         

  Deployment    Little or no     The approach   The approach is  The approach is  The approach is well   The approach is
                deployment of    is in the      deployed,        well deployed,   deployed, with no      fully deployed
                any systematic   early stages   although some    although         significant gaps.      without significant
                approach is      of deployment  areas or work    deployment may                          weaknesses or gaps
                evident.         in most areas  units are in     vary in some                            in any areas or
                                 or work units, early stages of  areas or work                           work units.
                                 inhibiting     deployment.      units.                                  
                                 progress in                                                             
                                 achieving the                                                           
                                 basic                                                                   
                                 requirements                                                            
                                 of the Item.                                                            

                                                                                                         

  Learning      An improvement   Early stages   The beginning of A fact-based,    Fact-based, systematic Fact-based,
                orientation is   of a           a systematic     systematic       evaluation and         systematic
                not evident;     transition     approach to      evaluation and   improvement and        evaluation and
                improvement is   from reacting  evaluation and   improvement      organizational         improvement and
                achieved through to problems to improvement of   process and some learning, including    organizational
                reacting to      a general      key processes is organizational   innovation, are key    learning through
                problems.        improvement    evident.         learning,        management tools;      innovation are key
                                 orientation                     including        there is clear         organization-wide
                                 are evident.                    innovation, are  evidence of refinement tools; refinement
                                                                 in place for     as a result of         and innovation,
                                                                 improving the    organizational-level   backed by analysis
                                                                 efficiency and   analysis and sharing.  and sharing, are
                                                                 effectiveness of                        evident throughout
                                                                 key processes.                          the organization.

                                                                                                         

  Integration   No               The approach   The approach is  The approach is  The approach is        The approach is
                organizational   is aligned     in the early     aligned with     integrated with        well integrated
                alignment is     with other     stages of        organizational   organizational needs   with organizational
                evident;         areas or work  alignment with   needs identified identified in response needs identified in
                individual areas units largely  basic            in response to   to the Organizational  response to the
                or work units    through joint  organizational   the              Profile and other      Organizational
                operate          problem        needs identified Organizational   Process Items.         Profile and other
                independently.   solving.       in response to   Profile and                             Process Items.
                                                the              other Process                           
                                                Organizational   Items.                                  
                                                Profile and                                              
                                                other Process                                            
                                                Items.                                                   

                                                                                                         
  ------------- ---------------- -------------- ---------------- ---------------- ---------------------- -------------------

**Guidance:** The overall score is not intended to be a numerical
average of the elements above; the Examiners select the range and score
that are most descriptive of the organization's achievement level for
the Item.

**Item 4.2---Overall Score**

> 0--5%
>
> 10--25%
>
> 30--45%
>
> 50--65% Item 4.2 Score %
>
> 70--85%
>
> 90--100%

**Item Worksheet---Item 7.1**

+-----------------------------------------------------------------------+
| **IMPORTANT! *Save this document using one of the following file      |
| names (as appropriate):***                                            |
|                                                                       |
| **[Your Role File Name]{.underline}**                                 |
|                                                                       |
| Item lead, posting version 1 Item 7_1v1                               |
|                                                                       |
| Item backup, posting changes to version 1 Item 7_1BU                  |
|                                                                       |
| Item lead, posting version 2 Item 7_1v2                               |
|                                                                       |
| Team member, posting feedback Item 7_1v2\_\[insert your initials\]    |
|                                                                       |
| Item lead, posting consensus-call-ready version Item 7_1v3            |
|                                                                       |
| Item lead, posting Consensus Scorebook version Item 7_1_consensus     |
+-----------------------------------------------------------------------+

> **Rationale for Item 7.1 Version**

**Instructions:** Summarize your rationale for selecting and
synthesizing the issues identified in your proposed comments (e.g.,
linkage to a critical key factor; a possible role-model practice or
result; an issue that may be preventing the applicant from scoring in a
higher range; a general deployment theme across most Independent Review
Worksheets; team members LJ, HH, DB, PS included this issue) and the
scoring range and score.

Note: This tool may be useful for summarizing your analysis before you
draft comments.

+-----+----------------------------------------------------------------+
| **S | **Summary of Proposed Strengths**                              |
| tre |                                                                |
| ngt | (Add or delete rows as needed. Aim for around 6 total combined |
| h** | strengths and opportunities for improvement \[OFIs\].)         |
|     |                                                                |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+
| 3   |                                                                |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
| *   | **Summary of Proposed Opportunities for Improvement**          |
| *OF |                                                                |
| I** | (Add or delete rows as needed. Aim for around 6 total combined |
|     | strengths and OFIs.)                                           |
| **\ |                                                                |
| #** |                                                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+
| 3   |                                                                |
+-----+----------------------------------------------------------------+

+-----+----------------------------------------------------------------+
|     | **Summary of KEY Observations Not Used**                       |
|     |                                                                |
|     | (Add or delete rows as needed.)                                |
+-----+----------------------------------------------------------------+
| 1   |                                                                |
+-----+----------------------------------------------------------------+
| 2   |                                                                |
+-----+----------------------------------------------------------------+

**Proposed Scoring Range and Score**

  ------------- ------------ ----------------------------------------------------
  **Process**   **Range      **Summarize below why this range is most
                selected**   appropriate.**

  Levels                     

  Trends                     

  Comparisons                

  Integration                
  ------------- ------------ ----------------------------------------------------

**For the Item as a whole, the range most representative of the
organization's achievement level is \_\_\_\_, with a score of
\_\_\_\_.**

Additional scoring justification not included above:

**Item Worksheet---Item 7.1**

Indicate the 4--6 most important key business/organization factors
relevant to this Item.

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

**Strengths and Opportunities for Improvement**

Indicate the relative importance/strength of the comment by bolding the
text for ++ or - - comments.

**STRENGTHS**

(Tab to move to the next column; tab from the final column to begin the
next comment.)

  ------------------------------------------------------------------------
  **Item   **STRENGTHS**
  Ref.**   
  -------- ---------------------------------------------------------------
           

  ------------------------------------------------------------------------

**OPPORTUNITIES FOR IMPROVEMENT**

*(Tab to move to the next column; tab from the final column to begin the
next comment.)*

  ------------------------------------------------------------------------
  **Item   **OPPORTUNITIES FOR IMPROVEMENT**
  Ref.**   
  -------- ---------------------------------------------------------------
           

  ------------------------------------------------------------------------

**Evaluation Factor Score Summary---Item 7.1**

+---------+--------+---------+----------+----------+---------+--------+
| **Guide | **0    | **10    | **3      | **5      | **70    | **90-- |
| lines** | --5%** | --25%** | 0--45%** | 0--65%** | --85%** | 100%** |
+---------+--------+---------+----------+----------+---------+--------+
| >       | There  | A few   | Good     | Good     | > Good  | Exc    |
|  Levels | are no | organiz | organi   | organi   | > to    | ellent |
|         | or     | ational | zational | zational | > ex    | or     |
|         | ganiza | perf    | per      | per      | cellent | ganiza |
|         | tional | ormance | formance | formance | >       | tional |
|         | perfo  | results | levels   | levels   | organiz | perfo  |
|         | rmance | are     | are      | are      | ational | rmance |
|         | r      | re      | reported | reported | > perf  | levels |
|         | esults | ported, | for some | for most | ormance | are    |
|         | and/or | and     | areas of | areas of | >       | re     |
|         | poor   | early   | im       | im       |  levels | ported |
|         | r      | good    | portance | portance | > are   | for    |
|         | esults | perf    | to the   | to the   | > r     | most   |
|         | in     | ormance | Item     | Item     | eported | areas  |
|         | areas  | levels  | requi    | requi    | > for   | of     |
|         | rep    | are     | rements. | rements. | > most  | impo   |
|         | orted. | evident |          |          | > areas | rtance |
|         |        | in a    |          |          | > of    | to the |
|         |        | few     |          |          | > imp   | Item   |
|         |        | areas.  |          |          | ortance | r      |
|         |        |         |          |          | > to    | equire |
|         |        |         |          |          | > the   | ments. |
|         |        |         |          |          | > Item  |        |
|         |        |         |          |          | >       |        |
|         |        |         |          |          |  requir |        |
|         |        |         |          |          | ements. |        |
+---------+--------+---------+----------+----------+---------+--------+
|         |        |         |          |          |         |        |
+---------+--------+---------+----------+----------+---------+--------+
| Trends  | Trend  | Some    | Some     | Be       | > Ben   | Bene   |
|         | data   | trend   | trend    | neficial | eficial | ficial |
|         | either | data    | data are | trends   | >       | trends |
|         | are    | are     | r        | are      |  trends | have   |
|         | not    | re      | eported, | evident  | > have  | been   |
|         | re     | ported, | and a    | in areas | > been  | sus    |
|         | ported | with    | majority | of       | > su    | tained |
|         | or     | some    | of the   | im       | stained | over   |
|         | show   | adverse | trends   | portance | > over  | time   |
|         | mainly | trends  | p        | to the   | > time  | in all |
|         | a      | e       | resented | accomp   | > in    | areas  |
|         | dverse | vident. | are      | lishment | > most  | of     |
|         | t      |         | ben      | of the   | > areas | impo   |
|         | rends. |         | eficial. | organi   | > of    | rtance |
|         |        |         |          | zation's | > imp   | to the |
|         |        |         |          | mission. | ortance | ac     |
|         |        |         |          |          | > to    | compli |
|         |        |         |          |          | > the   | shment |
|         |        |         |          |          | >       | of the |
|         |        |         |          |          | accompl | or     |
|         |        |         |          |          | ishment | ganiza |
|         |        |         |          |          | > of    | tion's |
|         |        |         |          |          | > the   | mi     |
|         |        |         |          |          | >       | ssion. |
|         |        |         |          |          | organiz |        |
|         |        |         |          |          | ation's |        |
|         |        |         |          |          | > m     |        |
|         |        |         |          |          | ission. |        |
+---------+--------+---------+----------+----------+---------+--------+
|         |        |         |          |          |         |        |
+---------+--------+---------+----------+----------+---------+--------+
| Comp    | Compa  | Little  | Early    | Some     | Many to | Ev     |
| arisons | rative | or no   | stages   | current  | most    | idence |
|         | infor  | comp    | of       | per      | trends  | of     |
|         | mation | arative | o        | formance | and     | in     |
|         | is not | info    | btaining | levels   | current | dustry |
|         | rep    | rmation | com      | have     | perf    | and    |
|         | orted. | is      | parative | been     | ormance | ben    |
|         |        | re      | inf      | e        | levels  | chmark |
|         |        | ported. | ormation | valuated | have    | lead   |
|         |        |         | are      | against  | been    | ership |
|         |        |         | evident. | relevant | ev      | is     |
|         |        |         |          | com      | aluated | demons |
|         |        |         |          | parisons | against | trated |
|         |        |         |          | and/or   | r       | in     |
|         |        |         |          | be       | elevant | many   |
|         |        |         |          | nchmarks | comp    | areas. |
|         |        |         |          | and show | arisons |        |
|         |        |         |          | areas of | and/or  |        |
|         |        |         |          | good     | ben     |        |
|         |        |         |          | relative | chmarks |        |
|         |        |         |          | perf     | and     |        |
|         |        |         |          | ormance. | show    |        |
|         |        |         |          |          | areas   |        |
|         |        |         |          |          | of      |        |
|         |        |         |          |          | lea     |        |
|         |        |         |          |          | dership |        |
|         |        |         |          |          | and     |        |
|         |        |         |          |          | very    |        |
|         |        |         |          |          | good    |        |
|         |        |         |          |          | r       |        |
|         |        |         |          |          | elative |        |
|         |        |         |          |          | perfo   |        |
|         |        |         |          |          | rmance. |        |
+---------+--------+---------+----------+----------+---------+--------+
|         |        |         |          |          |         |        |
+---------+--------+---------+----------+----------+---------+--------+
| Inte    | R      | Results | Results  | Organi   | Organiz | Or     |
| gration | esults | are     | are      | zational | ational | ganiza |
|         | are    | r       | reported | per      | perf    | tional |
|         | not    | eported | for many | formance | ormance | perfo  |
|         | re     | for a   | areas of | results  | results | rmance |
|         | ported | few     | im       | are      | are     | r      |
|         | for    | areas   | portance | reported | r       | esults |
|         | any    | of      | to the   | for most | eported | fully  |
|         | areas  | imp     | accomp   | key      | for     | a      |
|         | of     | ortance | lishment | cu       | most    | ddress |
|         | impo   | to the  | of the   | stomer/p | key     | key    |
|         | rtance | accompl | organi   | atient/\ | cu      | cus    |
|         | to the | ishment | zation's | student, | stomer, | tomer, |
|         | ac     | of the  | mission. | market,  | market, | m      |
|         | compli | organiz |          | and      | p       | arket, |
|         | shment | ation's |          | process  | rocess, | pr     |
|         | of the | m       |          | requi    | and     | ocess, |
|         | or     | ission. |          | rements. | action  | and    |
|         | ganiza |         |          |          | plan    | action |
|         | tion's |         |          |          | requir  | plan   |
|         | mi     |         |          |          | ements, | r      |
|         | ssion. |         |          |          | and     | equire |
|         |        |         |          |          | they    | ments, |
|         |        |         |          |          | include | and    |
|         |        |         |          |          | some    | they   |
|         |        |         |          |          | proj    | i      |
|         |        |         |          |          | ections | nclude |
|         |        |         |          |          | of      | proje  |
|         |        |         |          |          | future  | ctions |
|         |        |         |          |          | perfo   | of     |
|         |        |         |          |          | rmance. | future |
|         |        |         |          |          |         | perfor |
|         |        |         |          |          |         | mance. |
+---------+--------+---------+----------+----------+---------+--------+
|         |        |         |          |          |         |        |
+---------+--------+---------+----------+----------+---------+--------+

**Guidance:** The overall score is not intended to be a numerical
average of the elements above; the Examiners select the range and score
that are most descriptive of the organization's achievement level for
the Item.

**Item 7.1---Overall Score**

> 0--5%
>
> 10--25%
>
> 30--45%
>
> 50--65% Item 7.1 Score %
>
> 70--85%
>
> 90--100%

**OPPORTUNITIES FOR IMPROVEMENT**

***(Tab to move to the next column; tab from the final column to begin
the next comment.)***

Item Ref.