**2004**

**Consensus Manual**

**Revised July 2004**

> **KEY PROCESS CHANGES FOR 2004**

-   **Drafting and discussing site visit issues and reaching consensus
    on them at Stage 2 is optional. This step can be delayed until Stage
    3 (See Section Three).**

-   **A consensus hotline (1-800-898-4506) has been established to
    answer questions about the Criteria or the process during consensus
    calls. Team leaders must call the consensus hotline after each
    consensus call to report that the call has been completed.**

-   **While the NIST/BNQP monitor will participate in the entire
    planning call, the monitor may not listen to the entire consensus
    call. If you need assistance after the monitor leaves the call,
    contact the hotline.**

-   **Team leaders will complete evaluations of the consensus process.**

-   **This year's phone system will allow teams to plan and place calls
    without the assistance of an operator.**

**TABLE OF CONTENTS**

###### Section One: The Consensus Review Timeline 1 

###### 

#######  2004 Stage 2: Consensus Review Timeline 3

###### 

###### Section Two: Stage 2: Consensus Review Process Description 5 

###### 

#####  Purpose 7 

#####  Assignment of Consensus Team Members 7 

#####  The Consensus Review Process 7 

MBNQA Stage 2 Process (Flowchart) 8

###### 

###### Section Three: Stage 2: Roles and Responsibilities 11

#####  

#####  Consensus Team Members' Roles and Responsibilities (Matrix) 13 

Consensus Team Roles 14

##  Team Leader 14 

*Backup Team Leader* 14

*Scorebook Editor* 14

*Category/Item Lead* 14

*Category/Item Backup* 15

*Criteria Cop* 15

*Process Checker* 15

*Computer Expert* 15

*Scorebook Sponsor* 15

#####  Team Leader Responsibilities---Chronology of Activities 16

*Getting Started* 16

*Complete Planning and Prework* 16

*Draft Agenda for the Planning Call* 18

*Prior to the Consensus Calls* 19

*Conduct the Consensus Calls* 20

*Immediately Following the Consensus Calls* 21

Team Leader and Scorebook Editor Responsibilities 21

*The Consensus Scorebook* 21

#####  Maintaining Confidentiality 22 

#####  Planning Computer Use 23 

Options for Developing Site Visit Issues 24

Sample Peer Evaluation Letter 27

2004 Peer Evaluation Review Process 28

2004 Peer Evaluation Form 29

#####  

##### **Section Four: Sample Communications From ASQ** 31

> Letter to Team Leaders 32
>
> Letter to Backup Team Leaders 34
>
> Letter to Team Members 36

##### 

##### **Section Five: Communication Among Team Members** 39

#####  

#####  **Team Members to Team Leader** 41

#####  Dates Unavailable Form: Planning Call 42 

#####  Dates Unavailable Form: Consensus Calls 43 

#####  Biography Form 44 

Sample Completed Biography Form 45

#####  

#####  **Team Leader to Team Members** 46

#####  Sample Team Leader Planning Letter 47 

#####  Sample Planning Call Agenda 50 

#####  Sample Consensus Call Process 51 

#####  Sample Consensus Review Schedule 54 

#####  Sample Consensus Call Agendas 55 

> Sample Team Leader Thank You Note 56

# Section Six: Instructions, Samples, and Tools for the Consensus Process 57

#####  Steps for Creating a Draft Consensus Item Worksheet 59

#####  Synthesizing a Stage 2 Comment from Four Stage 1 Comments 60

#####  Sample Script---Presentation of Item 1.1 Summary 61

#####  Sample Item Worksheet 64

#####  Checklist for Process Checker to Use on Calls 66

#####  Rules for Consensus Scoring 67

#####  Tools Provided for Planning Consensus Assignments 68

#####  Stage 1 Examiner Scoring Table 68

#####  Stage 1 Scoring Graph 69

Stage 1 Frequency of Scores 70

> *Sample Stage 1 Examiner Scoring Table* 71
>
> *Sample Stage 1 Scoring Graph* 72
>
> *Sample Stage 1 Frequency of Scores* 73

#####  Preparing the Consensus Scorebook 74

**Section Seven: Materials From the Examiner Handbook and More FAQs** 77

**Index of Key Terms** 85

*Section One*

*The Consensus Review Timeline*

2004 Stage 2: Consensus Review Timeline

**July 29** Judges meeting---consensus applicants selected

**July 30--August 5** NIST/BNQP Award Process Team determines consensus
assignments and notifies ASQ

**August 2--3** The American Society for Quality **(**ASQ) sends team
members their assignments, team lists, available scorebooks, and general
consensus instructions

> ASQ sends applications to Examiners being added to a Consensus Team;
> Examiners return their Key Themes Worksheets to ASQ ***no later than
> August 17***

# By August 6 Team members send biographies and revised Dates Unavailable Forms to team leader

**By August 9** Team leader consults with team members to set conference
call dates and times

> Team leader notifies team, NIST/BNQP, and ASQ of conference call dates
> and times (at a minimum, the date and time of the planning call)

# By the scheduled Team leader and NIST/BNQP confer on scorebook editor and select editor; team leader 

# planning call forwards agenda for the planning call to the team

**Before all calls** ASQ sends team members information for joining the
conference calls. Team members confirm with ASQ receipt of call
information and notify ASQ of alternate contact information, if
necessary

# August 11--26 ASQ receives dates for consensus calls that have not previously been scheduled

**August 11--18** Planning calls

**August 17** Key Themes Worksheet(s) from new team members due to ASQ

**By August 19** ASQ sends new team members' Key Themes Worksheets to
all team members

**Aug. 30--Sept. 3** Primary and backup consensus calls for all teams

**September 7--8** Primary and backup consensus calls for all teams

**(Break for Labor**

**Day Weekend)**

# Immediately after Team leader calls the NIST/BNQP hotline (1-800-898-4506) to report that each

# consensus calls consensus call has ended. Team leader faxes consensus Score Summary Worksheet to ASQ after completing final call

#  

**24 hours after call** Team members send revised consolidated comments
and other assignments to

team leader/scorebook editor

# By September 13 Team leader sends (via Federal Express) feedback-ready consensus scorebook to arrive at NIST *no later than* *September 14* 

**September 17** Judges meet to select applicants that will receive a
site visit

***Section Two***

***Stage 2: Consensus Review***

***Process Description***

+---------------------+---+---------------------------------------------+
| # Purpose           |   | The purpose of Stage 2: Consensus Review is |
|                     |   | to clarify and resolve differences in       |
|                     |   | individual Examiners' review from Stage 1:  |
|                     |   | Independent Review. During consensus        |
|                     |   | review, a team of Examiners reaches         |
|                     |   | agreement on the applicant's strengths and  |
|                     |   | opportunities for improvement (OFIs), the   |
|                     |   | resulting score, and, if required by the    |
|                     |   | team leader, identifies 2--4 issues per     |
|                     |   | Criteria item to clarify and verify if the  |
|                     |   | applicant is selected for site visit. The   |
|                     |   | consensus scores are used to develop        |
|                     |   | scoring profiles for the Panel of Judges to |
|                     |   | use in its site visit selection process.    |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Assignment of       |   | A Consensus Team includes a team leader, a  |
| Consensus Team      |   | backup team leader, and a combination of    |
| Members             |   | Examiners (Senior, returning, new, and, in  |
|                     |   | some cases, alumni). Whenever possible,     |
|                     |   | team members are selected from Examiners    |
|                     |   | who completed the original Stage 1 review   |
|                     |   | of the application. Team leaders and backup |
|                     |   | team leaders are Senior Examiners who have  |
|                     |   | received additional training on the         |
|                     |   | Baldrige Award's purposes and processes.    |
|                     |   | One consensus team member will serve as the |
|                     |   | scorebook editor. Additional roles for team |
|                     |   | members include timekeeper, scorekeeper,    |
|                     |   | scorebook sponsor, Criteria cop, and        |
|                     |   | process checker.                            |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| The Consensus       |   | Consensus is a team decision about key      |
| Review Process      |   | factors, comments (Items and Key Themes),   |
|                     |   | and numerical scores, based on the          |
|                     |   | contributions of ALL Consensus Team members |
|                     |   | and all Stage 1 scorebooks, including those |
|                     |   | of Stage 1 Examiners who are not members of |
|                     |   | the Consensus Team. Via conference calls,   |
|                     |   | the Consensus Team reaches agreement on     |
|                     |   | comments synthesized from the Stage 1       |
|                     |   | scorebooks. The team also arrives at a      |
|                     |   | score for each Criteria Item (and reaches   |
|                     |   | consensus on site visit issues \[SVIs\], if |
|                     |   | appropriate). The scorebook editor prepares |
|                     |   | a consensus scorebook using the combined    |
|                     |   | work of the Consensus Team and forwards it  |
|                     |   | to the team leader for approval and         |
|                     |   | submission to NIST. The team leader is      |
|                     |   | ultimately responsible for ensuring that    |
|                     |   | the consensus scorebook is complete and is  |
|                     |   | submitted to NIST by the specified          |
|                     |   | deadline. The consensus scorebook is used   |
|                     |   | as the basis for the feedback report for    |
|                     |   | applicants not progressing to Stage 3, site |
|                     |   | visit review. It also serves as the basis   |
|                     |   | for site visit planning for applicants      |
|                     |   | advancing to Stage 3.                       |
+---------------------+---+---------------------------------------------+

![](media/image1.wmf)

![](media/image2.wmf)

***Section Three***

***Stage 2: Roles and Responsibilities***

+------+-------------------+-------------------+---------------------+
|      | **Planning and    | **Conducting the  | **Preparing the     |
|      | Prework**         | Consensus Call**  | Consensus           |
|      |                   |                   | Scorebook**         |
+------+-------------------+-------------------+---------------------+
| #    | -   Contacts team | -   Makes         | -   Coordinates and |
|  Tea |     members and   |     introductions |     confirms        |
| m Le |     introduces    |                   |     deadlines with  |
| ader |     self          | -   Establishes   |     team members    |
|      |                   |     ground rules  |     and scorebook   |
|      | -   Schedules     |                   |     editor          |
|      |     planning call | -   Verifies and  |                     |
|      |     and consensus |     clarifies     | -   Attaches Score  |
|      |     calls         |     process,      |     Summary         |
|      |                   |                   |     Worksheet       |
|      | -   Reviews Stage | responsibilities, |                     |
|      |     1 scoring     |     and schedule  | -   Sends paper     |
|      |     information   |                   |     copy and        |
|      |                   | -   Finalizes     |     electronic copy |
|      | -   Drafts        |     agenda        |     of scorebook to |
|      |     in            |                   |     NIST to arrive  |
|      | structions/ground | -   Discusses     |     no later than   |
|      |     rules for     |     computer use  |     September 14,   |
|      |     team and      |                   |     2004            |
|      |     agendas for   | -   Reviews order |                     |
|      |     calls         |     of discussion | -   Retains paper   |
|      |                   |     of Items      |     and electronic  |
|      | -   Drafts        |                   |     copy until      |
|      |     assignments   | -   Models        |     notified to     |
|      |     for team      |     presentation  |     destroy         |
|      |     (scorebook    |     style for     |                     |
|      |     editor,       |     team          | -   Prepares peer   |
|      |     Category/Item |                   |     evaluation      |
|      |     lead,         | -   Facilitates   |     forms           |
|      |     Category/Item |     discussion    |                     |
|      |     backup,       |                   |                     |
|      |     timekeeper,   | -   Recaps        |                     |
|      |     scorekeeper,  |     scoring       |                     |
|      |     scorebook     |     results       |                     |
|      |     sponsor,      |                   |                     |
|      |     computer      | -   Ensures calls |                     |
|      |     expert,       |     are completed |                     |
|      |     Criteria cop, |     by September  |                     |
|      |     process       |     8, 2004       |                     |
|      |     checker)      |                   |                     |
|      |                   | -   Calls the     |                     |
|      | -   Discusses     |     NIST/BNQP     |                     |
|      |     instructions, |     hotline       |                     |
|      |     plans for     |                   |                     |
|      |     calls, team   |  (1-800-898-4506) |                     |
|      |     assignments   |     to report     |                     |
|      |     (including    |     that each     |                     |
|      |     scorebook     |     consensus     |                     |
|      |     editor), and  |     call has      |                     |
|      |     agendas with  |     ended         |                     |
|      |     NIST/BNQP     |                   |                     |
|      |     contact       | -   Faxes Score   |                     |
|      |     point/monitor |     Summary       |                     |
|      |                   |     Worksheet to  |                     |
|      | -   Distributes   |     ASQ           |                     |
|      |     ins           |     immediately   |                     |
|      | tructions/agendas |     after call    |                     |
|      |     to team       |                   |                     |
|      |                   |                   |                     |
|      | -   Reviews       |                   |                     |
|      |     consensus     |                   |                     |
|      |     scoring and   |                   |                     |
|      |     rounding      |                   |                     |
|      |                   |                   |                     |
|      | -   Verifies      |                   |                     |
|      |     receipt of    |                   |                     |
|      |     team          |                   |                     |
|      |     information   |                   |                     |
|      |     by ASQ        |                   |                     |
|      |                   |                   |                     |
|      | -   Reviews Code  |                   |                     |
|      |     of            |                   |                     |
|      |     Conduct/Rules |                   |                     |
|      |     of Ethics     |                   |                     |
|      |                   |                   |                     |
|      | -   Develops plan |                   |                     |
|      |     for site      |                   |                     |
|      |     visit issues  |                   |                     |
+------+-------------------+-------------------+---------------------+
| **   | -   Review        | -   Provide input | -   Prepare final   |
| Team |     Criteria      |     on agenda     |     Item worksheets |
| M    |     requirements, |     topics        |                     |
| embe |     relevant Key  |                   | -   Forward final   |
| rs** |     Factors, and  | -   Serve as      |     Item worksheets |
|      |     guidelines    |     timekeeper,   |     to scorebook    |
|      |     for           |     scorekeeper,  |     editor, team    |
|      |     completing    |     scorebook     |     leader, and     |
|      |     Item          |     sponsor,      |     NIST Monitor    |
|      |     Worksheets    |     backup team   |                     |
|      |                   |     leader,       | -   Complete peer   |
|      | -   Serve as      |     process       |     evaluations and |
|      |     Category/Item |     checker,      |     forward to NIST |
|      |     lead and      |     Criteria cop, |                     |
|      |     Category/Item |     or computer   |                     |
|      |     backup and    |     expert, as    |                     |
|      |     review all    |     assigned      |                     |
|      |     Stage 1       |                   |                     |
|      |     scorebooks    | -   Lead          |                     |
|      |                   |     discussion on |                     |
|      | -   Complete Key  |     assigned      |                     |
|      |     Themes and    |     Items (review |                     |
|      |     submit        |     Criteria      |                     |
|      |     evaluation of |     requirements, |                     |
|      |     scorebooks    |     relevant key  |                     |
|      |     (new members) |     factors and   |                     |
|      |                   |     comments, and |                     |
|      | -   Develop draft |     propose       |                     |
|      |     Item          |     scoring range |                     |
|      |     Worksheets    |     and percent   |                     |
|      |                   |     score)        |                     |
|      | -   Distribute    |                   |                     |
|      |     draft         | -   Contribute to |                     |
|      |     worksheets to |     discussion of |                     |
|      |     other team    |     other Items   |                     |
|      |     members and   |     during the    |                     |
|      |     NIST Monitor  |     call          |                     |
|      |                   |                   |                     |
|      | -   Review other  | -   Arrive at     |                     |
|      |     team members' |     consensus on  |                     |
|      |     draft Item    |     comments and  |                     |
|      |     Worksheets    |     score         |                     |
|      |                   |                   |                     |
|      | -   Provide       | -   Record        |                     |
|      |     updated       |     information   |                     |
|      |     information   |     from          |                     |
|      |     to team       |     discussion on |                     |
|      |     leader        |     assigned      |                     |
|      |                   |     Items         |                     |
|      | -   Verify with   |                   |                     |
|      |     ASQ receipt   | -   Perform       |                     |
|      |     of conference |     assigned      |                     |
|      |     call forms    |     duties        |                     |
|      |     and provide   |                   |                     |
|      |     backup        | -   Provide input |                     |
|      |     contact       |     for Key       |                     |
|      |     information   |     Themes        |                     |
|      |                   |     Worksheet     |                     |
|      | -   Review Code   |                   |                     |
|      |     of            | -   Recap scoring |                     |
|      |     Conduct/Rules |     results on    |                     |
|      |     of Ethics, if |     assigned      |                     |
|      |     necessary     |     Items         |                     |
+------+-------------------+-------------------+---------------------+
| **S  | -   May           | -   Contributes   | -   Coordinates     |
| core |     p             |     to the        |     with            |
| book | repare/distribute |     discussion of |     Category/Item   |
| Edit |     to team Key   |     Items during  |     leads to ensure |
| or** |     Factors and   |     the call      |     comments and    |
|      |     Key Themes    |                   |     site visit      |
| *    |     Worksheets,   | -   Arrives at    |     issues (if      |
| *(*C |     if asked by   |     consensus on  |     applicable)     |
| ould |     team leader   |     comments and  |     reflect the     |
| be   |                   |     score         |     calls'          |
| the  | -   Reviews other |                   |     Discussion and  |
| team |     team members' | -   Ensures all   |     Criteria        |
| lea  |     draft Item    |     comments are  |     requirements    |
| der, |     Worksheets    |     well-written  |                     |
| ba   |     and provides  |     and conform   | -   Compiles        |
| ckup |     comments      |     to comment    |     consensus Item  |
| team |                   |     guidelines    |     worksheets      |
| lea  | -   Provides      |                   |                     |
| der, |     updated       | -   Captures      | -   Reviews Item    |
| or   |     information   |     discussion of |     worksheets to   |
| ano  |     to team       |     Items for     |     eliminate       |
| ther |     leader        |     inclusion in  |     conflicts and   |
| team |                   |     Scorebook     |     develop         |
| me   | -   Prepares      |                   |     linkages        |
| mber |     evaluation    | -   Finalizes Key |                     |
| *)** |     forms         |     Themes and    | -   Finalizes Key   |
|      |                   |     Key Factors   |     Factors         |
|      | -   Verifies with |     Worksheets,   |     Worksheet       |
|      |     ASQ receipt   |     if assigned   |                     |
|      |     of conference |     by the TL     | -   Finalizes Key   |
|      |     call forms    |                   |     Themes          |
|      |     and provides  |                   |     Worksheet       |
|      |     backup        |                   |                     |
|      |     contact       |                   | -   Formats the     |
|      |     information   |                   |     scorebook       |
|      |                   |                   |                     |
|      | \- Reviews Code   |                   |                     |
|      | of Conduct        |                   |                     |
+------+-------------------+-------------------+---------------------+
| *    | -   Forms         | -   Explains role | -   Provides        |
| *NIS |     consensus     |     of NIST       |     assistance on   |
| T/BN |     teams         |     monitor to    |     process         |
| QP** |                   |     team members  |                     |
|      | -   Sends team    |                   | -   Receives        |
|      |     assignments   | -   Monitors      |     consensus       |
|      |     to ASQ        |     planning      |     scorebook       |
|      |                   |     call(s) and   |                     |
|      | -   Monitors      |     start of      | -   Edits the       |
|      |     process,      |     consensus     |     feedback report |
|      |     answers       |     call(s)       |                     |
|      |     questions     |                   | -   Prepares for    |
|      |                   | -   Provides      |     Judges' meeting |
|      | -   Supports team |     assistance on |                     |
|      |     leaders in    |     process and   |                     |
|      |     planning      |     Criteria      |                     |
|      |     consensus     |                   |                     |
|      |     calls         | -   Staffs the    |                     |
|      |                   |     consensus     |                     |
|      | -   Assigns and   |     hotline       |                     |
|      |     notifies new  |                   |                     |
|      |     team members  |  (1-800-898-4506) |                     |
|      |                   |                   |                     |
|      | \- Confers with   |                   |                     |
|      | team leader to    |                   |                     |
|      | assign scorebook  |                   |                     |
|      | editor            |                   |                     |
+------+-------------------+-------------------+---------------------+
| **A  | -   Distributes   | -   Provides      | -   Prepares        |
| SQ** |     team lists    |     support for   |     Judges'         |
|      |                   |                   |     materials       |
|      | -   Provides team | telecommunication |                     |
|      |     member        |     issues        |                     |
|      |     information   |                   |                     |
|      |     to team       | -   Collects      |                     |
|      |     leader        |     completed     |                     |
|      |                   |     Score Summary |                     |
|      | -   Forwards      |     Worksheet     |                     |
|      |     consensus     |     from team     |                     |
|      |     packages to   |     leader        |                     |
|      |     teams         |                   |                     |
+------+-------------------+-------------------+---------------------+
| **   | -   Review the    |                   | -   Select          |
| Judg |     instructional |                   |     applicants for  |
| es** |     materials and |                   |     site visits     |
|      |     samples prior |                   |                     |
|      |     to meeting    |                   |                     |
+------+-------------------+-------------------+---------------------+

+---------------------+---+---------------------------------------------+
| Consensus Team      |   | The team leader has several roles during    |
| Roles---            |   | the consensus review process:               |
|                     |   |                                             |
| Team Leader         |   | 1.  *Plans the consensus review process for |
|                     |   |     the Consensus Team, guides the team     |
|                     |   |     through the key steps of the process,   |
|                     |   |     and monitors the process for both       |
|                     |   |     progress and quality*                   |
|                     |   |                                             |
|                     |   | 2.  *Develops a plan for completing site    |
|                     |   |     visit issues (SVIs) in the event the    |
|                     |   |     applicant is selected for a site visit  |
|                     |   |     (See "Options for Developing Site Visit |
|                     |   |     Issues, page 24.)*                      |
|                     |   |                                             |
|                     |   | 3.  *Instructs the Consensus Team in any    |
|                     |   |     changes to the consensus review process |
|                     |   |     not covered during the Examiner         |
|                     |   |     Preparation course*                     |
|                     |   |                                             |
|                     |   | 4.  *Serves as a mentor and role model for  |
|                     |   |     Senior Examiners who have not led       |
|                     |   |     and/or participated in Stage 2, as well |
|                     |   |     as for new and returning Examiners*     |
|                     |   |                                             |
|                     |   | 5.  *Models the desired process for         |
|                     |   |     presenting assigned Categories/Items*   |
|                     |   |                                             |
|                     |   | 6.  *Reviews and approves the consensus     |
|                     |   |     scorebook prior to its submission to    |
|                     |   |     NIST*                                   |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | In addition to his/her roles as a Consensus |
| Roles---Backup Team |   | Team member, the backup team leader works   |
| Leader              |   | with the team leader to plan the consensus  |
|                     |   | process. The backup also assists the team   |
|                     |   | leader in communicating with the other team |
|                     |   | members. ***Most importantly, the backup    |
|                     |   | assumes the team leader role if the team    |
|                     |   | leader is unable to fulfill the role.***    |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| #### C              |   | The team leader assigns the scorebook       |
| onsensus Team Roles |   | editor with assistance from NIST. The       |
| ---Scorebook Editor |   | editor is responsible for working with all  |
|                     |   | Category/Item leads to ensure that all      |
|                     |   | comments are well-written and conform to    |
|                     |   | the comment guidelines. The scorebook       |
|                     |   | editor may be responsible for drafting the  |
|                     |   | Key Themes Worksheet and the Key Factors    |
|                     |   | Worksheet as part of the prework before the |
|                     |   | calls.                                      |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | Individual team members act as leads on     |
| Ro                  |   | specific Items and/or Categories assigned   |
| les---Category/Item |   | to them by the team leader. A significant   |
| Lead                |   | amount of prework must be completed prior   |
|                     |   | to the start of the consensus calls. The    |
| Consensus Team      |   | Category/Item lead role is to ensure that   |
| Ro                  |   |                                             |
| les---Category/Item |   | 1.  ***ALL** Scorebooks are reviewed for    |
| Lead                |   |     the assigned Items*                     |
|                     |   |                                             |
| *-continued*        |   | 2.  *draft Item Worksheets are prepared     |
|                     |   |     with key factors, strengths, and        |
|                     |   |     opportunities for improvement*          |
|                     |   |                                             |
|                     |   | 3.  *areas of disagreement that account for |
|                     |   |     scoring differences among Examiners are |
|                     |   |     captured*                               |
|                     |   |                                             |
|                     |   | 4.  *a proposed scoring range based on the  |
|                     |   |     comments and appropriate scoring        |
|                     |   |     guidelines is included*                 |
|                     |   |                                             |
|                     |   | 5.  *2--4 key SVIs per Item are drafted (if |
|                     |   |     applicable)*                            |
|                     |   |                                             |
|                     |   | 6.  *the draft Item Worksheets are sent to  |
|                     |   |     the Category/Item backup and the NIST   |
|                     |   |     monitor*                                |
|                     |   |                                             |
|                     |   | 7.  *after the backup's review, the Item    |
|                     |   |     worksheets are revised and then sent to |
|                     |   |     all team members and the NIST monitor*  |
|                     |   |                                             |
|                     |   | 8.  *a peer review form for each scorebook  |
|                     |   |     is completed and returned to NIST*      |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | The primary role of the Category/Item       |
| Role---             |   | backup is to review the proposed draft Item |
|                     |   | worksheets prepared by the Category/Item    |
| Category/Item       |   | lead to ensure the comments address all     |
| Backup              |   | major points of the Criteria Items without  |
|                     |   | going beyond the Criteria requirements.     |
|                     |   | Also, the backup should suggest changes     |
|                     |   | and/or additions to the comments based on   |
|                     |   | the comment guidelines. ***Most             |
|                     |   | importantly, the backup should be prepared  |
|                     |   | to lead the discussion if the Examiner      |
|                     |   | assigned as the lead is unable to           |
|                     |   | participate.***                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | The primary role of the Criteria cop is to  |
| Roles---Criteria    |   | ensure that the consensus call discussions  |
| Cop                 |   | and comments are limited to what is         |
|                     |   | specified in the Criteria.                  |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | The primary role of the process checker is  |
| Roles---Process     |   | to ensure that the team discussions of each |
| Checker             |   | Item follow the process outlined in the     |
|                     |   | consensus manual, (i.e., Criteria, key      |
|                     |   | factors, ***s***trengths/OFIs, comment      |
|                     |   | differences causing scoring differences,    |
|                     |   | proposed scoring range, score, and site     |
|                     |   | visit issues, if applicable.) See Section   |
|                     |   | Six for Checklist for Process Checker to    |
|                     |   | Use on Calls                                |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | The primary role of the computer expert is  |
| Roles---Computer    |   | to provide guidance to team members on file |
| Expert              |   | labeling conventions and to resolve         |
|                     |   | hardware- and software-related issues. (See |
|                     |   | instructions for planning computer use on   |
|                     |   | page 23.)                                   |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | The scorebook sponsor represents the views  |
| Roles---            |   | expressed in the scorebook(s) of Stage 1    |
|                     |   | Examiners not participating in the          |
| Scorebook Sponsor   |   | consensus process (sometimes referred to as |
|                     |   | "phantom scorebooks"). The sponsor          |
|                     |   | represents these views in the consensus     |
|                     |   | Item Worksheets and corresponding           |
|                     |   | discussions.                                |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
| Consensus Team      |   | The timekeeper ensures that the consensus   |
| Roles---            |   | discussions adhere to the timeframes set in |
|                     |   | the agenda and records the duration of each |
| #### Timekeeper     |   | call.                                       |
+---------------------+---+---------------------------------------------+

+---------------------+---+---------------------------------------------+
| **Team Leader       |   |  Review the consensus package sent by ASQ. |
| Responsibilities--- |   | It will contain                             |
| Chronology of       |   |                                             |
| Activities**        |   | ##### *a list of team members*              |
|                     |   |                                             |
| **Getting Started** |   | ###                                         |
|                     |   | ## *Stage 1: Independent review scorebooks* |
|                     |   |                                             |
|                     |   | 1.  *scoring profiles (Stage 1 Examiner     |
|                     |   |     Scoring Table, Stage 1 Scoring Graph,   |
|                     |   |     and Stage 1 Frequency of Scores)*       |
|                     |   |                                             |
|                     |   | 2.  *Dates Unavailable Forms*               |
|                     |   |                                             |
|                     |   | #####  Review the                          |
|                     |   | *Consensus Manual* and any on-line Stage 2  |
|                     |   |                                             |
|                     |   | #####  materials                            |
|                     |   |                                             |
|                     |   |  Review the Consensus Roles and            |
|                     |   | Responsibilities Matrix                     |
|                     |   |                                             |
|                     |   |  Contact the team members immediately      |
|                     |   | after the list of                           |
|                     |   |                                             |
|                     |   | consensus team members is received and      |
|                     |   |                                             |
|                     |   | ##### *introduce you                        |
|                     |   | rself and provide your biographical form.*  |
|                     |   |                                             |
|                     |   | ##### *ask for biogr                        |
|                     |   | aphies and updated Dates Unavailable Forms* |
|                     |   |                                             |
|                     |   | 1.  *confirm all team members' telephone,   |
|                     |   |     fax (identify whether secure or not     |
|                     |   |     secure), and address data from          |
|                     |   |     mid-August through mid-September*       |
|                     |   |                                             |
|                     |   | 2.  *instruct Examiners to notify you if    |
|                     |   |     they do not receive the mailing from    |
|                     |   |     ASQ or if it is incomplete or           |
|                     |   |     incorrect*                              |
|                     |   |                                             |
|                     |   | 3.  *solicit each Examiner's Item interest  |
|                     |   |     and expertise for preliminary           |
|                     |   |     discussion of team assignments*         |
|                     |   |                                             |
|                     |   | 4.  *survey team members about their        |
|                     |   |     computer and software access, reminding |
|                     |   |     them that all materials should be       |
|                     |   |     prepared using the NIST requested       |
|                     |   |     format (Times New Roman, 12 point) and  |
|                     |   |     saved in Microsoft Word 6.0/95*         |
+---------------------+---+---------------------------------------------+
|                     |   | #####                                       |
+---------------------+---+---------------------------------------------+
|                     |   | ##### If there is                           |
|                     |   | a new team member, determine when his/her c |
|                     |   | ompleted Key Themes Worksheet will be retur |
|                     |   | ned to ASQ ---must be no later than **Augus |
|                     |   | t 17**. Reviewing and/or discussing any app |
|                     |   | licant information (e.g., draft Key Factors |
|                     |   |  and/or Key Themes Worksheets, strengths/OF |
|                     |   | Is, or scorebooks) may not occur until ASQ  |
|                     |   | receives and distributes the additional Key |
|                     |   |  Themes Worksheet(s) from new team members. |
+---------------------+---+---------------------------------------------+

+---------------------+---+---+--------------------------------------------+---+---+
| Team Leader         |   |   |  Discuss the possible scorebook editor    |   |   |
|                     |   |   | assignment with NIST/BNQP, taking into     |   |   |
| Responsibilities--- |   |   | account writing skill, willingness, and    |   |   |
|                     |   |   | availability of potential editors. The     |   |   |
| Chronology of       |   |   | scorebook editor can be the team leader,   |   |   |
| Activities          |   |   | backup team leader, or another member of   |   |   |
|                     |   |   | the team. Typically, the scorebook editor  |   |   |
| Complete Planning   |   |   | assignment is given to Examiners who have  |   |   |
| and                 |   |   | at least one year of prior consensus       |   |   |
|                     |   |   | experience.                                |   |   |
| Prework             |   |   |                                            |   |   |
|                     |   |   |  Contact the identified scorebook editor  |   |   |
| Team Leader         |   |   | and discuss the role.                      |   |   |
|                     |   |   |                                            |   |   |
| Responsibilities--- |   |   |  Determine Consensus Team assignments by  |   |   |
|                     |   |   |                                            |   |   |
| Chronology of       |   |   | 1.  *reviewing team members' scorebooks*   |   |   |
| Activities          |   |   |                                            |   |   |
|                     |   |   | 2.  *evaluating Item expertise, knowledge  |   |   |
| Complete Planning   |   |   |     of the Criteria, quality of comments*  |   |   |
| and                 |   |   |                                            |   |   |
|                     |   |   | 3.  *considering the Examiners' preference |   |   |
| Prework             |   |   |     for an Item*                           |   |   |
|                     |   |   |                                            |   |   |
| *- continued*       |   |   | 4.  *assessing tentative Consensus Team    |   |   |
|                     |   |   |     assignments: Category/Item leads,      |   |   |
|                     |   |   |     Category/Item backups, scorebook       |   |   |
|                     |   |   |     editor, scorebook sponsor for phantom  |   |   |
|                     |   |   |     scorebooks, etc.*                      |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | #####                                      |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
|                     |   |   |  Determine and assign the pairings of     |   |   |
|                     |   |   | lead and back-up responsibilities.         |   |   |
|                     |   |   | Consider balancing the following factors:  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 1.  *an Examiner experienced in the MBNQA  |   |   |
|                     |   |   |     process with an inexperienced          |   |   |
|                     |   |   |     Examiner*                              |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 2.  *an introvert with an extrovert*       |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 3.  *good writing skills with adequate     |   |   |
|                     |   |   |     writing skills.*                       |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
|                     |   |   |  Schedule the consensus conference calls  |   |   |
|                     |   |   | (planning and consensus calls):            |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 1.  *Use the updated Dates Unavailable     |   |   |
|                     |   |   |     Forms to select call dates and times   |   |   |
|                     |   |   |     consistent with Examiners'             |   |   |
|                     |   |   |     availability*                          |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 2.  *Select dates and times for a one hour |   |   |
|                     |   |   |     planning call (August 11--18)*         |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 3.  *Select dates and times for all        |   |   |
|                     |   |   |     remaining consensus calls, which must  |   |   |
|                     |   |   |     be held during the timeframe specified |   |   |
|                     |   |   |     on the consensus timeline (August      |   |   |
|                     |   |   |     30--September 3 and September 7--8).*  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 4.  *Contact each team member with the     |   |   |
|                     |   |   |     proposed call dates and times and      |   |   |
|                     |   |   |     confirm availability*                  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 5.  *Notify NIST/BNQP and ASQ of agreed    |   |   |
|                     |   |   |     upon dates and times*                  |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
|                     |   |   |  Develop a tentative schedule for the     |   |   |
|                     |   |   | consensus review from initial preparation  |   |   |
|                     |   |   | through delivery of the final, completed   |   |   |
|                     |   |   | consensus scorebook. Key dates should      |   |   |
|                     |   |   | include                                    |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 1.  *call dates and times(for planning     |   |   |
|                     |   |   |     call and two consensus calls)*         |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 2.  *draft Item worksheets due to          |   |   |
|                     |   |   |     Category/Item backups, team leader,    |   |   |
|                     |   |   |     NIST/BNQP monitor*                     |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 3.  *revisions of draft Item Worksheets*   |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 4.  *revised Item worksheets and other     |   |   |
|                     |   |   |     assignments due to full team and       |   |   |
|                     |   |   |     NIST/BNQP monitor*                     |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 5.  *completion of assignments after the   |   |   |
|                     |   |   |     call*                                  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 6.  *completion of the consensus scorebook |   |   |
|                     |   |   |     by the scorebook editor; review and    |   |   |
|                     |   |   |     approval of final consensus scorebook  |   |   |
|                     |   |   |     by the team leader; and overnight      |   |   |
|                     |   |   |     delivery via Federal Express of the    |   |   |
|                     |   |   |     consensus scorebook to NIST (see       |   |   |
|                     |   |   |     Consensus Review Timeline)*            |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
| Team Leader         |   |   |  Develop a plan for completion/discussion |   |   |
|                     |   |   | of SVIs. (See "Options for Developing Site |   |   |
| Responsibilities--- |   |   | Visit Issues," page 24.)                   |   |   |
|                     |   |   |                                            |   |   |
| Chronology of       |   |   |  Draft correspondence to team members     |   |   |
| Activities          |   |   | containing assignments, instructions,      |   |   |
|                     |   |   | expectations, ground rules, and agendas    |   |   |
| Complete Planning   |   |   | for the team. (See "Draft Agenda for the   |   |   |
| and                 |   |   | Planning Call," below, and sample          |   |   |
|                     |   |   | correspondence and agendas in Section      |   |   |
| Prework             |   |   | Five, pages 47--55*.*)                     |   |   |
|                     |   |   |                                            |   |   |
| *- continued*       |   |   |  Review the tentative action plan,        |   |   |
|                     |   |   | consensus strategy, SVI plan, and draft    |   |   |
|                     |   |   | agendas for the calls, as well as          |   |   |
|                     |   |   | correspondence, with the NIST/BNQP         |   |   |
|                     |   |   | monitor.                                   |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | -   Plan to model the desired process,     |   |   |
|                     |   |   |     using a less complicated Category      |   |   |
|                     |   |   |     (usually Items from Category 1         |   |   |
|                     |   |   |     Leadership).                           |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   Plan to discuss corresponding Items in |   |   |
|                     |   |   |     Category 7 with the applicable Process |   |   |
|                     |   |   |     Items.                                 |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | -   Distribute the action plan, consensus  |   |   |
|                     |   |   |     strategy, SVI plan, and agendas for    |   |   |
|                     |   |   |     the calls, as well as correspondence,  |   |   |
|                     |   |   |     to team members and the NIST/BNQP      |   |   |
|                     |   |   |     monitor.                               |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   |  Confirm with ASQ that all Examiners have |   |   |
|                     |   |   | acknowledged receipt of the conference     |   |   |
|                     |   |   | call date/time information.                |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   |  Distribute team biographies and          |   |   |
|                     |   |   | consensus call information prior to the    |   |   |
|                     |   |   | call.                                      |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
| Team Leader         |   |   | The planning call can be used for three    |   |   |
|                     |   |   | purposes:                                  |   |   |
| Responsibilities--- |   |   |                                            |   |   |
|                     |   |   | 1.  *a team-building function to help the  |   |   |
| Chronology of       |   |   |     team get acquainted*                   |   |   |
| Activities          |   |   |                                            |   |   |
|                     |   |   | 2.  *a planning function, allowing the     |   |   |
| Draft Agenda for    |   |   |     team leader to cover the logistics of  |   |   |
| the                 |   |   |     the consensus review process with all  |   |   |
|                     |   |   |     team members at the same time*         |   |   |
| Planning Call       |   |   |                                            |   |   |
|                     |   |   | 3.  *possibly beginning the scorebook      |   |   |
| Team Leader         |   |   |     review process. Some teams are able to |   |   |
|                     |   |   |     begin discussing draft Key Factors and |   |   |
| Responsibilities--- |   |   |     Key Themes Worksheets during the       |   |   |
|                     |   |   |     planning call if all new team members  |   |   |
| Chronology of       |   |   |     have sent their Key Themes Worksheets  |   |   |
| Activities          |   |   |     to ASQ and if all team members are on  |   |   |
|                     |   |   |     the call for the entire discussion.*   |   |   |
| Draft Agenda for    |   |   |                                            |   |   |
| the                 |   |   |  Topics to cover in the planning call     |   |   |
|                     |   |   | include the following (See sample agenda   |   |   |
| Planning Call       |   |   | in Section Five):                          |   |   |
|                     |   |   |                                            |   |   |
| *-continued*        |   |   | 1.  *a roll call: the operator will        |   |   |
|                     |   |   |     conduct a roll call to ensure all team |   |   |
|                     |   |   |     members are present.*                  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 2.  *the reconnection policy: review the   |   |   |
|                     |   |   |     procedure for reconnecting if a team   |   |   |
|                     |   |   |     member is disconnected, and remind the |   |   |
|                     |   |   |     team NOT to disconnect during breaks.* |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 3.  *introductions:*                       |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | -   *Begin the process of matching         |   |   |
|                     |   |   |     Examiners' names with their voices.*   |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *NIST/BNQP monitor explains his/her    |   |   |
|                     |   |   |     role throughout the consensus          |   |   |
|                     |   |   |     process.*                              |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 4.  *a check to ensure all working         |   |   |
|                     |   |   |     materials have been received*          |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 5.  *ground rules for the calls*           |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 6.  *assignments (e.g., scorebook editor,  |   |   |
|                     |   |   |     Category/Item leads, Category/Item     |   |   |
|                     |   |   |     backups, scorekeeper, timekeeper,      |   |   |
|                     |   |   |     scorebook sponsors, computer expert,   |   |   |
|                     |   |   |     Criteria cop, and process checker)*    |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 7.  *an overview of preparation            |   |   |
|                     |   |   |     requirements*                          |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 8.  *a consensus review schedule*          |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 9.  *consensus call dates and times*       |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 10. *an agenda for the remaining consensus |   |   |
|                     |   |   |     calls*                                 |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 11. *confidentiality procedures (e.g.,     |   |   |
|                     |   |   |     secure/unsecure fax machines, computer |   |   |
|                     |   |   |     use and files, e-mails, etc.)*         |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 12. *scorebook preparation software,       |   |   |
|                     |   |   |     formatting, and distribution           |   |   |
|                     |   |   |     procedures*                            |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 13. *completion of peer evaluation forms*  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 14. *Site visit availability*              |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 15. *optional planning call topics: draft  |   |   |
|                     |   |   |     Key Factors Worksheet, draft Key       |   |   |
|                     |   |   |     Themes Worksheet (only if new team     |   |   |
|                     |   |   |     members have sent Key Themes Worksheet |   |   |
|                     |   |   |     to ASQ and all team members are        |   |   |
|                     |   |   |     present for the discussion)*           |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
| Team Leader         |   |   | Prior to the remaining consensus call(s),  |   |   |
|                     |   |   | ensure that                                |   |   |
| Responsibilities--- |   |   |                                            |   |   |
|                     |   |   | 1.  *all Examiners have used the correct   |   |   |
| Chronology of       |   |   |     Criteria (Business, Health Care, or    |   |   |
| Activities          |   |   |     Education) in their evaluations*       |   |   |
|                     |   |   |                                            |   |   |
| Prior to the        |   |   | 2.  *all Examiners have completed their    |   |   |
| Consensus Calls     |   |   |     draft Item Worksheets and any other    |   |   |
|                     |   |   |     assignments. The backup team leader    |   |   |
| Team Leader         |   |   |     can assist with this task.*            |   |   |
|                     |   |   |                                            |   |   |
| Responsibilities--- |   |   | 3.  *all Item Worksheets have been         |   |   |
|                     |   |   |     reviewed and commented on by the       |   |   |
| Chronology of       |   |   |     assigned Category/Item backups*        |   |   |
| Activities          |   |   |                                            |   |   |
|                     |   |   | 4.  *all revised Item Worksheets and other |   |   |
| Prior to the        |   |   |     assigned work have been distributed to |   |   |
| Consensus Calls     |   |   |     all team members and the NIST/BNQP     |   |   |
|                     |   |   |     monitor*                               |   |   |
| *-continued*        |   |   |                                            |   |   |
|                     |   |   | 5.  *team members have reviewed all        |   |   |
|                     |   |   |     documents for each Item and are ready  |   |   |
|                     |   |   |     to serve as scorebook sponsors, as     |   |   |
|                     |   |   |     assigned*                              |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 6.  *all Examiners know to connect with    |   |   |
|                     |   |   |     the conference call several minutes    |   |   |
|                     |   |   |     before the call is to begin*           |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 7.  *a copy of all communications,         |   |   |
|                     |   |   |     documents, and materials have been     |   |   |
|                     |   |   |     sent to all team members and to the    |   |   |
|                     |   |   |     NIST/BNQP monitor*                     |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 8.  *a discussion of the consensus call    |   |   |
|                     |   |   |     agenda has occurred with the NIST/BNQP |   |   |
|                     |   |   |     monitor*                               |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 9.  *the consensus call agenda has been    |   |   |
|                     |   |   |     distributed to all team members and    |   |   |
|                     |   |   |     the NIST/BNQP monitor*                 |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
| Team Leader         |   |   | The calls include the following topics     |   |   |
|                     |   |   | (See sample agenda in Section Five):       |   |   |
| Responsibilities--- |   |   |                                            |   |   |
|                     |   |   | 1.  *a roll call by the operator*          |   |   |
| #### Chron          |   |   |                                            |   |   |
| ology of Activities |   |   | 2.  *instructions if                       |   |   |
|                     |   |   |     disconnected/procedures when on break* |   |   |
| **Conduct the       |   |   |                                            |   |   |
| Consensus Calls**   |   |   | 3.  *various roles (e.g., timekeeper,      |   |   |
|                     |   |   |     scorekeeper, process checker, etc.)*   |   |   |
| #                   |   |   |                                            |   |   |
|                     |   |   | 4.  *the role of the NIST/BNQP monitor*    |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 5.  *agenda review*                        |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 6.  *ground rules*                         |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 7.  *rules for consensus scoring (see      |   |   |
|                     |   |   |     Section Six)*                          |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 8.  *Key Factors Worksheet*                |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 9.  *Key Themes Worksheet review (the team |   |   |
|                     |   |   |     members discuss their initial          |   |   |
|                     |   |   |     impressions of the applicant's key     |   |   |
|                     |   |   |     themes)*                               |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 10. *the Item discussions, modeled by the  |   |   |
|                     |   |   |     team leader, including*                |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | -   *criteria Item requirements*           |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *4--6 key factors for the Item*        |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *a review of strengths and OFIs*       |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | -   *discussion of areas of agreement and  |   |   |
|                     |   |   |     differences*                           |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *discussion of unique comments*        |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *consensus on strengths and OFIs*      |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *a review of Scoring Guidelines*       |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *consensus on scoring range*           |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | -   *consensus on SVIs (if applicable)*    |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *assignment of a percentage score      |   |   |
|                     |   |   |     within an agreed-upon range*           |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
|                     |   |   | > *11. review and confirmation of Item     |   |   |
|                     |   |   | > scores*                                  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 12. *Key Themes Worksheet review and       |   |   |
|                     |   |   |     finalization*                          |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 13. *scoring results recap; review Score   |   |   |
|                     |   |   |     Summary Worksheet*                     |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 14. *next steps review. Confirm schedules  |   |   |
|                     |   |   |     and deadlines for*                     |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | -   *revisions*                            |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *exchange of materials*                |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *submission of materials*              |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | ```{=html}                                 |   |   |
|                     |   |   | <!-- -->                                   |   |   |
|                     |   |   | ```                                        |   |   |
|                     |   |   | -   *production and submission of the      |   |   |
|                     |   |   |     final scorebook*                       |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | 15. *review of site visit schedule;        |   |   |
|                     |   |   |     confirm Examiners' availability*       |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
| Team Leader         |   |   |  Call the NIST/BNQP hotline               |   |   |
|                     |   |   | (1-800-898-4506) after each consensus call |   |   |
| Responsibilities--- |   |   | to report that it has ended.               |   |   |
|                     |   |   |                                            |   |   |
| ####### **Chronol   |   |   |  Fax the Score Summary Worksheet to ASQ   |   |   |
| ogy of Activities** |   |   | (1-414-765-7214) after the final call,     |   |   |
|                     |   |   | only.                                      |   |   |
| Immediately         |   |   |                                            |   |   |
| Following the       |   |   |                                            |   |   |
| Consensus Calls     |   |   |                                            |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
| **Team Leader and   |   |   | The consensus review process results in a  |   |   |
| Scorebook Editor    |   |   | consensus scorebook. NIST reformats the    |   |   |
| Re                  |   |   | final consensus scorebook into a feedback  |   |   |
| sponsibilities---** |   |   | report, which is presented to the          |   |   |
|                     |   |   | applicant if it does not progress to Stage |   |   |
| **The Consensus     |   |   | 3: Site Visit Review.                      |   |   |
| Scorebook**         |   |   |                                            |   |   |
|                     |   |   |  Responsibility                           |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | The team leader is ultimately responsible  |   |   |
|                     |   |   | for the preparation, the content quality,  |   |   |
|                     |   |   | and the timely submission of the team's    |   |   |
|                     |   |   | consensus scorebook with feedback-ready    |   |   |
|                     |   |   | comments.                                  |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | The team writes the consensus scorebook    |   |   |
|                     |   |   | collaboratively. That is, the              |   |   |
|                     |   |   | Category/Item leads are responsible for    |   |   |
|                     |   |   | preparing the Item Worksheets, and team    |   |   |
|                     |   |   | members are responsible for reviewing and  |   |   |
|                     |   |   | agreeing upon final comments.              |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | The scorebook editor completes the Key     |   |   |
|                     |   |   | Factors and Key Themes Worksheets and      |   |   |
|                     |   |   | edits the Item and Key Worksheets to       |   |   |
|                     |   |   | ensure they meet the Comment Guidelines.   |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | The team leader reviews all worksheets to  |   |   |
|                     |   |   | ensure that they have been revised to      |   |   |
|                     |   |   | reflect the discussion on the consensus    |   |   |
|                     |   |   | call and has the ultimate responsibility   |   |   |
|                     |   |   | to review and approve the quality of the   |   |   |
|                     |   |   | scorebook.                                 |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   |  Timely Submission                        |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | The final, edited, and feedback-ready      |   |   |
|                     |   |   | consensus scorebook (paper copy and        |   |   |
|                     |   |   | diskette/CD) must arrive at NIST by the    |   |   |
|                     |   |   | specified deadline (see the Consensus      |   |   |
|                     |   |   | Review Timeline in Section One)**.** The   |   |   |
|                     |   |   | team leader and scorebook editor retain a  |   |   |
|                     |   |   | copy of all materials sent to NIST (both   |   |   |
|                     |   |   | paper copy and diskette/CD) until notified |   |   |
|                     |   |   | by ASQ that the feedback report has been   |   |   |
|                     |   |   | sent to the applicant. This will           |   |   |
|                     |   |   | facilitate any discussions concerning the  |   |   |
|                     |   |   | content during the NIST editorial process. |   |   |
|                     |   |   | If the applicant proceeds to Stage 3, the  |   |   |
|                     |   |   | consensus scorebook will serve as the      |   |   |
|                     |   |   | starting point for site visit planning.    |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | In order to meet the deadline for          |   |   |
|                     |   |   | submission to NIST, team leaders must      |   |   |
|                     |   |   | allow time following the consensus call(s) |   |   |
|                     |   |   | for revisions by Category/Item leads,      |   |   |
|                     |   |   | editing and/or formatting                  |   |   |
+---------------------+---+---+--------------------------------------------+---+---+
| **Team Leader and   |   |   | by the scorebook editor, and review and    |   |   |
| Scorebook Editor    |   |   | approval by the team leader.               |   |   |
| Re                  |   |   |                                            |   |   |
| sponsibilities---** |   |   | The team leader's careful planning process |   |   |
|                     |   |   | during the preparation stage can minimize  |   |   |
| **The Consensus     |   |   | last-minute rework. Planning time for the  |   |   |
| Scorebook** *-      |   |   | team leader to review the draft worksheets |   |   |
| continued*          |   |   | before the consensus call allows him/her   |   |   |
|                     |   |   | to guide the Category/Item leads during    |   |   |
|                     |   |   | the revision periods both preceding and    |   |   |
|                     |   |   | following the call(s). Emphasizing the     |   |   |
|                     |   |   | role of the Category/Item backup to review |   |   |
|                     |   |   | the draft worksheets and ensuring time for |   |   |
|                     |   |   | this step before the consensus call will   |   |   |
|                     |   |   | also reduce rework during the final        |   |   |
|                     |   |   | preparation steps. Stressing the need for  |   |   |
|                     |   |   | simple, uniform word processing formats    |   |   |
|                     |   |   | and high-quality draft Item Worksheets in  |   |   |
|                     |   |   | the earliest steps of the consensus        |   |   |
|                     |   |   | process will make the scorebook editor's   |   |   |
|                     |   |   | job more efficient at the editing stage.   |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   |  Preparation                              |   |   |
|                     |   |   |                                            |   |   |
|                     |   |   | Section Six of this manual contains        |   |   |
|                     |   |   | additional, detailed instructions for      |   |   |
|                     |   |   | producing the consensus scorebook.         |   |   |
|                     |   |   | Detailed, step-by-step instructions for    |   |   |
|                     |   |   | scorebook editors will be mailed following |   |   |
|                     |   |   | the Judges' meeting.                       |   |   |
+---------------------+---+---+--------------------------------------------+---+---+

+---------------------+---+---------------------------------------------+
| Maintaining         |   | During consensus, it is crucial that all    |
| Confidentiality     |   | Award materials remain confidential,        |
|                     |   | including all printed materials, faxed      |
| Maintaining         |   | materials, e-mails, and computer files. In  |
| Confidentiality     |   | addition, an Examiner cannot ask another    |
|                     |   | individual who is not a Consensus Team      |
| *-continued*        |   | member to transcribe written documents      |
|                     |   | relating to the Award application           |
|                     |   | evaluation. The application and all         |
|                     |   | Examiner evaluation materials should be     |
|                     |   | kept in a secure location when not in use.  |
|                     |   | E-mail may be used in general planning      |
|                     |   | (e.g., for distribution of agendas or       |
|                     |   | biographies); however, no MBNQA evaluation  |
|                     |   | materials may be sent by e-mail due to the  |
|                     |   | difficulty of securing Internet             |
|                     |   | communications. Likewise, if a fax machine  |
|                     |   | is used to exchange drafts and comments     |
|                     |   | among Examiners, confidentiality must be    |
|                     |   | maintained. MBNQA evaluation materials may  |
|                     |   | be faxed **only** if the Examiner receiving |
|                     |   | the fax has a **secure fax** or is at the   |
|                     |   | fax machine to **receive the material as it |
|                     |   | arrives.** The sending Examiner must call   |
|                     |   | the receiving Examiner before faxing the    |
|                     |   | material to verify that he/she can go to    |
|                     |   | the fax machine to receive the confidential |
|                     |   | materials. **This material must be marked   |
|                     |   | "confidential."** Evaluation materials also |
|                     |   | may be exchanged by overnight delivery.     |
|                     |   |                                             |
|                     |   | Maintaining the confidentiality of the      |
|                     |   | Award Program requires that computer        |
|                     |   | (electronic) files be treated with the same |
|                     |   | degree of security as paper copies of Award |
|                     |   | application materials. When not in use,     |
|                     |   | electronic files should be removed from the |
|                     |   | computer hard drive and stored on a clearly |
|                     |   | marked diskette that is placed in a secure  |
|                     |   | location with the written application       |
|                     |   | materials (e.g., a locked file or file      |
|                     |   | cabinet). Electronic files containing Award |
|                     |   | evaluations should never be placed on a     |
|                     |   | computer hard drive or diskette where       |
|                     |   | anyone other than the Examiner has access   |
|                     |   | to the file.                                |
|                     |   |                                             |
|                     |   | Many word processing software systems and   |
|                     |   | some computer systems automatically create  |
|                     |   | a backup file. Each Examiner needs to       |
|                     |   | determine if a backup file is automatically |
|                     |   | created, and, if so, where it is located    |
|                     |   | and what it is named. Backing up files is   |
|                     |   | important and Examiners are urged to do so. |
|                     |   | Examiners also are urged to be certain that |
|                     |   | backup files are treated with the same      |
|                     |   | degree of security as the working files.    |
+---------------------+---+---------------------------------------------+

+---------------------+---+---------------------------------------------+
|                     |   | After the consensus calls are completed and |
|                     |   | changes incorporated into the drafts, each  |
|                     |   | Examiner should send both a paper copy and  |
|                     |   | a copy of the associated electronic files   |
|                     |   | on diskette for each section to the team    |
|                     |   | leader and scorebook editor. Once the       |
|                     |   | Examiner has verified that both the team    |
|                     |   | leader and scorebook editor have received   |
|                     |   | both copies of each section, the Examiner   |
|                     |   | should remove all electronic files          |
|                     |   | (including backup files) from the hard      |
|                     |   | drive, store the files on a diskette, and   |
|                     |   | keep the diskette in a secure location. If  |
|                     |   | the applicant does not go on to receive a   |
|                     |   | site visit, each Examiner will need to      |
|                     |   | return all Award materials to ASQ and to    |
|                     |   | delete the associated electronic files. It  |
|                     |   | is then desirable to reformat the diskettes |
|                     |   | so the file/information cannot be retrieved |
|                     |   | using software recovery programs.           |
|                     |   |                                             |
|                     |   | If the applicant is selected for a site     |
|                     |   | visit, the Site Visit Team will coordinate  |
|                     |   | computer use.                               |
+---------------------+---+---------------------------------------------+

+---------------------+---+---------------------------------------------+
| **Planning Computer |   | In the team's planning call, discuss        |
| Use**               |   | computer use, and, if possible, identify    |
|                     |   | someone on the team to coordinate it and    |
| **Planning Computer |   | ensure that there will be compatibility     |
| Use**               |   | among the computer files generated by the   |
|                     |   | team members.                               |
| *-continued*        |   |                                             |
|                     |   | -   Develop a plan to produce compatible    |
|                     |   |     computer output (files). This should    |
|                     |   |     include                                 |
|                     |   |                                             |
|                     |   | 1.  *the word processing software version   |
|                     |   |     and diskette or CD format to save       |
|                     |   |     shared files (e.g., Word 6.0/95).       |
|                     |   |     Macintosh users should convert the file |
|                     |   |     to a PC-compatible file.*               |
|                     |   |                                             |
|                     |   | 2.  *a file naming convention*              |
|                     |   |                                             |
|                     |   | 3.  *document format requirements:*         |
|                     |   |                                             |
|                     |   | ####### *fo                                 |
|                     |   | nt: Times New Roman and 12 point type size* |
|                     |   |                                             |
|                     |   | -   *margins: 1"*                           |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *page numbering and location*           |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *line spacing: single space comments    |
|                     |   |     and double space between comments*      |
|                     |   |                                             |
|                     |   | 4.  *page format/layout issues*             |
|                     |   |                                             |
|                     |   | 5.  *format for Item Worksheets*            |
|                     |   |                                             |
|                     |   | -   *Item number on the first line*         |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *key factors (4--6)*                    |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *Text-only comments, i.e., no bold,     |
|                     |   |     underlining, colors, etc.*              |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *Use tabs for spacing between*          |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *left page margin and + or --*          |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *+ or -- and a, b, c*                   |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   *a, b, c and first word of the comment* |
|                     |   |                                             |
|                     |   |  Determine which team member is            |
|                     |   | responsible for each scorebook worksheet.   |
|                     |   |                                             |
|                     |   |  Ensure each Examiner has the most current |
|                     |   | version of virus detection software and     |
|                     |   | ensure each Examiner checks all diskettes,  |
|                     |   | CDs, and the hard drive for computer        |
|                     |   | viruses.                                    |
|                     |   |                                             |
|                     |   |  Ensure that all files have up-to-date     |
|                     |   | backups created.                            |
|                     |   |                                             |
|                     |   |  Ensure that backup files are kept on      |
|                     |   | separate diskettes/CDs and that             |
|                     |   | write-protect tabs are in place.            |
|                     |   |                                             |
|                     |   |  If mailing diskettes or CDs, use a        |
|                     |   | protective mailer for transit.              |
+---------------------+---+---------------------------------------------+

+---------------------+---+---------------------------------------------+
| **Options for       |   | Each Site Visit Team must draft and reach   |
| Developing Site     |   | consensus on 2--4 SVIs per Item that the    |
| Visit Issues**      |   | team will clarify or verify when on site.   |
|                     |   | The Consensus Team leader has the           |
| **Options for       |   | discretion to decide when each step of the  |
| Developing Site     |   | SVI development process (drafting,          |
| Visit Issues**      |   | reviewing, and revising SVIs) will take     |
|                     |   | place and who will participate.             |
| *-continued*        |   |                                             |
|                     |   | 1.  Options for **WHEN** SVIs are           |
| **Options for       |   |     developed:                              |
| Developing Site     |   |                                             |
| Visit Issues**      |   | -   **The traditional process occurs during |
|                     |   |     Stage 2**. Each Item lead drafts 2--4   |
| *-continued*        |   |     SVIs per Item during the consensus      |
|                     |   |     planning phase. The drafts are reviewed |
|                     |   |     by Item backups and then revised by the |
|                     |   |     Item leads before the consensus calls.  |
|                     |   |     During the consensus calls, the team    |
|                     |   |     discusses and reaches consensus on all  |
|                     |   |     SVIs for all Items. Use of this         |
|                     |   |     traditional process is optional. Team   |
|                     |   |     leaders who are relatively confident    |
|                     |   |     that their consensus applicant will     |
|                     |   |     move on to a site visit may wish to     |
|                     |   |     continue this process. Others may wish  |
|                     |   |     to wait until they are certain that     |
|                     |   |     their applicant is moving on to         |
|                     |   |     Stage 3. The disadvantage of waiting is |
|                     |   |     that it will increase the amount of     |
|                     |   |     work the site visit team must           |
|                     |   |     accomplish during the Stage 3 planning  |
|                     |   |     phase.                                  |
|                     |   |                                             |
|                     |   | -   **Between the Stage 2 and Stage 3       |
|                     |   |     (September\                             |
|                     |   |     14--17).** After the submission of the  |
|                     |   |     team's Stage 2 scorebook, there are a   |
|                     |   |     few days before the Judges' meeting     |
|                     |   |     (where decisions are made concerning    |
|                     |   |     which applicants will receive a site    |
|                     |   |     visit). Team leaders can use this time  |
|                     |   |     for the development of SVIs. Possible   |
|                     |   |     options for who should prepare the SVIs |
|                     |   |     are discussed below.                    |
|                     |   |                                             |
|                     |   | -   **Early in the Stage 3 planning         |
|                     |   |     phase**. Team Leaders may wait to       |
|                     |   |     develop SVIs until they learn if their  |
|                     |   |     applicant will receive a site visit.    |
|                     |   |     This decision will be made at the       |
|                     |   |     Judges' meeting on September 17.        |
|                     |   |     Waiting until this time will increase   |
|                     |   |     the amount of work that must be         |
|                     |   |     accomplished during the site visit      |
|                     |   |     planning phase, but it will guarantee   |
|                     |   |     that the SVIs will be used. Possible    |
|                     |   |     options for who should prepare the SVIs |
|                     |   |     are discussed below.                    |
|                     |   |                                             |
|                     |   | 2.  Options for **WHO** participates in the |
|                     |   |     phases of SVI development**:**          |
|                     |   |                                             |
|                     |   | -   **Drafting SVIs**: The Team leader may  |
|                     |   |     draft all the SVIs, have the backup     |
|                     |   |     team leader draft all of the SVIs, have |
|                     |   |     Item leads draft SVIs for their         |
|                     |   |     assigned Items, or assign a combination |
|                     |   |     of team members to draft them.          |
|                     |   |                                             |
|                     |   | ```{=html}                                  |
|                     |   | <!-- -->                                    |
|                     |   | ```                                         |
|                     |   | -   **Reviewing Draft SVIs**: The Team      |
|                     |   |     leader will assign one or more members  |
|                     |   |     of the team to review the SVIs and give |
|                     |   |     feedback to the drafter(s). The         |
|                     |   |     reviewers can be one or more team       |
|                     |   |     members (e.g., team leader, backup team |
|                     |   |     leader, the Item backups from the       |
|                     |   |     consensus stage, Senior Examiners on    |
|                     |   |     the team).                              |
|                     |   |                                             |
|                     |   | -   **Revising Draft SVIs**: The original   |
|                     |   |     drafter(s) will revise the SVIs based   |
|                     |   |     on feedback from the reviewer(s).       |
|                     |   |                                             |
|                     |   | **Reaching Consensus on SVIs**---The entire |
|                     |   | team must reach consensus on all the SVIs.  |
|                     |   | This can be done during the consensus       |
|                     |   | calls, the period between Stages 2 and 3,   |
|                     |   | or during the early part of the Stage 3     |
|                     |   | (site visit) planning process.              |
|                     |   |                                             |
|                     |   | **The use of the Item Worksheet and the SVI |
|                     |   | Worksheet---**SVIs should appear on the     |
|                     |   | appropriate Item Worksheet in the section   |
|                     |   | labeled **"Site Visit Issues (For Stage 3,  |
|                     |   | Site Visit Use)."**                         |
|                     |   |                                             |
|                     |   | If the applicant is selected for a site     |
|                     |   | visit, each Item/Category lead will copy    |
|                     |   | and paste each SVI onto a separate SVI      |
|                     |   | Worksheet and develop a strategy (e.g.,     |
|                     |   | documents to review, people to interview,   |
|                     |   | questions to ask) for resolving the issue   |
|                     |   | while on site.                              |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+
|                     |   |                                             |
+---------------------+---+---------------------------------------------+

**SAMPLE PEER EVALUATION LETTER**

MEMORANDUM

TO: Consensus Team Members

FROM: Award Process Team Members

SUBJECT: Peer Evaluation - 2004 Scorebooks

We are again requesting your cooperation in providing peer feedback by
evaluating the 2004 scorebooks for your consensus application.

Attached you will find instructions for reviewing scorebooks and a form
for recording your assessment. Please complete one of the enclosed Peer
Evaluation Forms for each Stage 1 Scorebook that you have received.

Your evaluation should be based on the value of the scorebook to you in
preparing your consensus drafts. Please be constructive in your
comments.

The evaluation forms will be forwarded to the Examiner who prepared the
scorebook to provide him/her with feedback on the value of his or her
work to fellow Examiners. Since Examiners have asked for feedback on
their performance, your assessment will be appreciated.

Please complete the forms after you have prepared your draft consensus
assignment. Return the forms to Bob Fangmeyer at NIST in the enclosed
envelope marked "*Peer Evaluations"* after your consensus call (by
September 15, if possible).

Thank you for your willingness to provide feedback to your fellow
Examiners.

**2004 Peer Evaluation Review Process**

(Completed by Category/Item Leads and/or Scorebook Editors)

Materials Needed

1.  *Criteria for Performance Excellence* for applicable sector
    (Business, Education, or Health Care)

2.  Comment Guidelines

3.  Scorebooks or worksheets for the assigned applicant

4.  Blank 2004 Peer Evaluation Forms

Preparation

1.  Complete the header information on each review sheet: scorebook
    prepared by (Examiner name), applicant #, reviewed by (your name),
    and date (that you are doing the evaluations).

2\. Complete one Peer Evaluation Form for each 2004 scorebook that you
used in your work.

Review and Rate Scorebooks

Respond to the following statements either with a yes/no or with a
rating from 1 (poor) to 5 (excellent).

*1. Key Factors Worksheet*

-   Check to be sure that the worksheet is a bulleted list of the most
    significant facts/characteristics for the applicant and is correctly
    divided into five sections (organizational environment,
    organizational relationships, competitive environment, strategic
    challenges, and performance improvement system).

*2. Item Worksheets*

-   Check to be sure that you have a complete set of Item Worksheets (19
    for business, health care and education).

-   Scan the Item Worksheet pages and check that the six to ten comments
    *visually approximate* the score; for example, you might see four
    strengths and four OFIs for a score of 50%.

-   Scan the Item Worksheet pages and check to see that the number of
    comments is reasonably balanced across all Categories; for example,
    no strength or OFI section of an Item Worksheet is blank, or no one
    worksheet or set of worksheets has an excessive number of comments.

-   On Results Items, check to see that comments identified gaps, levels
    and trends, segmentation, and the appropriate comparisons.

*3. Comment Guidelines*

-   Rate the Examiner's compliance with each of the Comment Guidelines.

-   Circle specific problems from the list provided in item 3e on the
    evaluation form.

*4. Key Themes Worksheet*

-   Check that the comments answer the three Key Themes questions (most
    important strengths, significant opportunities, and significant
    results) and that the worksheet presents an overall summary of the
    key points of the application evaluation, taking into account Item
    linkages, key factors, and core values.

*5. Overall Scorebook Quality*

-   Considering all of your responses to the above guidelines, rate the
    overall scorebook quality.

6\. *Comments*

-   Add any additional feedback (positive or developmental) that would
    be helpful to the original Examiner author in his/her future
    scorebook work.

Return your completed 2004 Scorebook Review forms to Bob Fangmeyer at
NIST in the enclosed envelope.

2004 Peer Evaluation Form

+--------------------------------------------------------+---+---+---+---+---+---+
| # Scorebook Guidelines                                 | > | > | > | > | > | > |
|                                                        |   |   |   |   |   |   |
| Please check your role:                                | P | * | * | * | * | * |
|                                                        | o | * | * | * | * | * |
| ####### Stage 1 Scor                                   | o | F | G | A | E | N |
| ebook Editor \_\_\_\_\_ Stage 2 Category Lead \_\_\_\_ | r | a | o | b | x | o |
|                                                        |   | i | o | o | c | t |
| Stage 2 Scorebook Editor \_\_\_\_                      |   | r | d | v | e | > |
|                                                        |   | * | * | e | l |   |
| **This review will be shared with the Examiner whose   |   | * | * | > | l | A |
| Scorebook you are evaluating**                         |   |   |   |   | e | p |
|                                                        |   |   |   | A | n | p |
|                                                        |   |   |   | v | t | l |
|                                                        |   |   |   | e | * | i |
|                                                        |   |   |   | r | * | c |
|                                                        |   |   |   | a |   | a |
|                                                        |   |   |   | g |   | b |
|                                                        |   |   |   | e |   | l |
|                                                        |   |   |   | * |   | e |
|                                                        |   |   |   | * |   | * |
|                                                        |   |   |   |   |   | * |
+--------------------------------------------------------+---+---+---+---+---+---+
| **1. Key Factors Worksheet---**Concisely presents a    | 1 | 2 | 3 | 4 | 5 |   |
| bulleted list of the most significant facts and        |   |   |   |   |   |   |
| characteristics of the applicant\'s organization.      |   |   |   |   |   |   |
| Divided into five sections using the headings named in |   |   |   |   |   |   |
| the "Preface: Organizational Profile."                 |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| **2. Item Worksheets**                                 |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| a\. Appropriate number of strengths/OFIs for the Item  | 1 | 2 | 3 | 4 | 5 |   |
| score                                                  |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| b\. Number of comments balanced across Categories      | 1 | 2 | 3 | 4 | 5 |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| c\. On Results Items, comments identified gaps, levels | 1 | 2 | 3 | 4 | 5 |   |
| and trends, segmentation, and the appropriate          |   |   |   |   |   |   |
| comparisons                                            |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| **3. Comment Guidelines**                              |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| a\. Clearly specifies the strengths and OFIs in a      | 1 | 2 | 3 | 4 | 5 |   |
| single complete thought                                |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| b\. Comments address central requirements of the       | 1 | 2 | 3 | 4 | 5 |   |
| Criteria                                               | 1 |   |   |   |   |   |
|                                                        | 1 |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| c\. Comments reflect the most relevant and important   | 1 | 2 | 3 | 4 | 5 |   |
| key factors for the applicant                          |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| ### d. Comments draw linkages across It                | 1 | 2 | 3 | 4 | 5 |   |
| ems, or between an Item and the Organizational Profile |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| e.  Comments are "feedback ready." If not, circle the  | 1 | 2 | 3 | 4 | 5 |   |
|     following specific problem(s):                     |   |   |   |   |   |   |
|                                                        |   |   |   |   |   |   |
| Prescriptive Judgmental Beyond Criteria Conflicting +  |   |   |   |   |   |   |
| and OFI                                                |   |   |   |   |   |   |
|                                                        |   |   |   |   |   |   |
| Parrots application Too general Uses jargon Critical   |   |   |   |   |   |   |
| tone                                                   |   |   |   |   |   |   |
|                                                        |   |   |   |   |   |   |
| Acronyms not used by applicant Vague                   |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| **4. Key Themes Worksheet---**Presents an overall      | 1 | 2 | 3 | 4 | 5 |   |
| summary of the key points of the application           |   |   |   |   |   |   |
| evaluation, taking into account important Item         |   |   |   |   |   |   |
| strengths, OFIs, key factors, and core values          |   |   |   |   |   |   |
+--------------------------------------------------------+---+---+---+---+---+---+
| **5.** **Overall Scorebook Quality**                   | 1 | 2 | 3 | 4 | 5 |   |
+--------------------------------------------------------+---+---+---+---+---+---+

### 

### Please provide a comment on the overall quality of the Scorebook:\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Please provide an example for problems noted in item 3e above:

***Section Four***

***Sample Communications From ASQ***

# LETTER TO TEAM LEADERS

Dear *Team Leader*:

Thank you for agreeing to lead the Consensus Team for #XX-00. The
success of the consensus process is critically linked to how well the
team prepares for, participates in, and follows up after the consensus
calls. As team leader, you are responsible for planning and leading the
consensus process, mentoring the team, and ensuring the timely
completion of the final consensus scorebook.

Preparation is a critical aspect of the consensus process; please review
the enclosed *Consensus Manual* and Quickstart Instructions for Team
Leaders. They are also available on-line at
[www.baldrige.nist.gov/Examiner_Resources.htm](http://www.baldrige.nist.gov/Examiner_Resources.htm).
Please pay particular attention to the initial steps in the planning
process, e.g., scheduling calls, preparing instructions for the team.
Planning calls must occur between August 11 and August 18; consensus
calls must be held August 30 through September 3 and September 7 and 8.

All team members who read the application at Stage 1 will receive
scoring data and scorebooks prepared by the Stage 1 Examiners, including
the Examiners who have not moved on to Stage 2. *New team members,
[including new team leaders]{.underline}, must evaluate the application,
develop Key Theme Worksheets, and fax the worksheets to ASQ
(414-765-7214**)** by **August 17, 2004***. ASQ will forward the
worksheets to all team members when it receives them. New team members
will receive copies of all scoring data and completed scorebooks when
they complete their Key Themes Worksheets.

**[Enclosures]{.underline}:** The instructions and materials will vary
depending on whether you are an original read to the team (you read the
application at Stage 1) or you are a new read to the team (you did not
read the application at Stage 1). On the page that follows, find the
check box that corresponds to your situation, and ensure that you have
all pertinent materials. Please call the ASQ Baldrige Department
immediately at 414-765-7205 if you are missing any materials.

**[Contacting your team]{.underline}:** Contact your team members by
**August 4, 2004,** to introduce yourself and to discuss the schedule of
calls and tentative team assignments. In addition, send your completed
biography form to your team by August 4.

**[Protecting confidentiality]{.underline}** of the applicant is
paramount and a challenge during the consensus process.

-   In your **written work or in discussions** with your team members,
    refer to the applicant only by its number, as "the organization," or
    as "the applicant."

-   When **faxing** materials to your team members, the receiving fax
    machine must be secure, or the recipient must be available to
    personally retrieve the fax as it is being sent.

-   **E-mail** may NOT be used for consensus materials unless the
    materials are purely administrative (e.g., agendas, ground rules for
    the calls), and they contain no mention of the applicant.

-   **Phones** must be secure---cellular and portable phones are **NOT**
    secure.

**[Problems:]{.underline}** If your schedule has changed and you are no
longer available to participate in consensus, or if you have any
concerns or questions, please contact:

Debbie Smyth 301-975-5312 Bob Fangmeyer 301-975-4781

Sandra Byrne 301-975-4812 Mike Berry 301-975-8312

Mark Shapiro 301-975-3621

** [TEAM LEADER---ORIGINAL READ:]{.underline}**

\_\_\_\_ You will be leading an INTACT team. All Examiners read and
scored the application in Stage 1.

\_\_\_\_ One or more members of your team are new to the application in
Stage 2: Consensus Review. New team members have until **August 17,
2004**, to return their completed Key Themes Worksheets to ASQ. ASQ will
then forward the Key Themes Worksheets to you and all team members.

ENCLOSED:

  --- ------------------------------- ---- ---------------------------------
  1   Urgent Reply Form **(Fax,       7    Federal Express package to return
      phone, or e-mail ASQ                 final consensus scorebook to NIST
      immediately upon receipt of          **by September 14, 2004**
      package)**                           

  2   Blank Biography Form            8    Stage 1: Independent Review
      (**complete and send to team         scorebooks *except for new team
      members by August 4, 2004, if        member(s)*
      possible)**                          

  3   Team Roster                     9    Scoring data (*Stage 1 Examiner
                                           Scoring Table, Stage 1 Scoring
                                           Graph, Stage 1 Frequency Table)*

  4   Quickstart Instructions for     10   Scoring Guidelines with Comment
      Team Leaders                         Guidelines on reverse side

  5   Consensus Manual                11   Dates Unavailable Forms completed
                                           by team members

  6   Scorebook cover                 12   Peer Evaluation Forms and
                                           envelopes to return to NIST
  --- ------------------------------- ---- ---------------------------------

** [TEAM LEADER---NEW READ]{.underline}**

As a new read to this team, **you must complete and return your Key
Themes Worksheet to ASQ no later than August 17, 2004. Your Key Themes
Worksheet may be faxed to ASQ at 414-765-7214.**

\_\_\_\_ You will be leading an INTACT team. All Examiners read and
scored the application in Stage 1.

\_\_\_\_ One or more members of your team are new to the application in
Stage 2: Consensus Review. New team members have until **August 17,
2004,** to return their completed Key Themes Worksheets to ASQ. ASQ will
then forward the Key Themes Worksheets to you and all team members.

ENCLOSED:

  --- ------------------------------- ---- ---------------------------------
  1   Urgent Reply Form **(Fax,       8    Federal Express package to return
      phone, or e-mail ASQ                 Final Consensus Scorebook to NIST
      immediately to confirm receipt       **by September 14, 2004**
      of package**)                        

  2   Application (with               9    Stage 1: Independent Review
      Considerations for Reviewing         scorebooks
      Small Businesses, if                 
      applicable)                          

  3   Blank Biography Form            10   Scoring data (*Stage 1 Examiner
      **(complete and send to team         Scoring Table, Stage 1 Scoring
      members by August 4, 2004, if        Graph, Stage 1 Frequency Table)*
      possible**)                          

  4   Team Roster                     11   Scoring Guidelines with Comment
                                           Guidelines on reverse side

  5   Quickstart Instructions for     12   Dates Unavailable Forms completed
      Team Leaders                         by team members

  6   Consensus Manual                13   Peer Evaluation Forms and
                                           envelopes to return to NIST

  7   Scorebook cover                 14   Procedures and Federal Express
                                           envelope/label for returning
                                           materials to ASQ at the
                                           completion of Award cycle
  --- ------------------------------- ---- ---------------------------------

Thank you for the commitment of your time during the 2004 Award cycle.

Peter LaBonte, MBNQA Program

Enclosuure

LETTER TO BACKUP TEAM LEADERS

Dear *Backup Team Leader*:

Thank you for agreeing to be backup team leader for Consensus Team
#0XX-00. As backup, you will work with the team leader to plan the
consensus process. The backup also assists the team leader in
communicating with the other team members. Most importantly, the backup
assumes the team leader role if the team leader is unable to fulfill the
role.

Since preparation is such a critical aspect of the consensus process,
please review the enclosed *Consensus Manual* and the Quickstart
Instructions for Team Leaders. They are also available on-line at
[www.baldrige.nist.gov/Examiner_Resources.htm](http://www.baldrige.nist.gov/Examiner_Resources.htm).
Please pay particular attention to the initial steps in the planning
process, e.g., scheduling calls, preparing instructions for the team.
Planning calls must occur between August 11 and August 18; consensus
calls must be held August 30 through September 3 and September 7 and 8.

**[Enclosures]{.underline}:** The instructions and materials will vary
depending on whether you are an original read to the team (you read the
application at Stage 1) or you are a new read to the team (you did not
read the application at Stage 1). On the page that follows, please find
the check box that corresponds to your situation, and ensure that you
have the materials listed under that check box. Please call the ASQ
Baldrige Department immediately at 414-765-7205 if you are missing any
materials. *New team members must evaluate the application, develop Key
Theme Worksheets, and fax the worksheets to ASQ (414-765-7214**)** by
**August 17, 2004***. ASQ will forward the worksheets to each team
member when it receives them. New team members will receive copies of
all scoring data and completed scorebooks when they complete their Key
Themes Worksheets.

**[Team Leader Contact:]{.underline}** The team leader will contact you
by August 4, 2004, to discuss the consensus schedule and tentative team
assignments. Remember that the team leader is trying to coordinate the
schedule of six team members, so please be as flexible as possible.

**[Protecting confidentiality]{.underline}** of the applicant is
paramount and a challenge during the consensus process.

-   In your **written work or in discussions** with your team members,
    refer to the applicant only by its number, as "the organization," or
    as "the applicant."

-   When **faxing** materials to your team members, the receiving fax
    machine must be secure or the recipient must be available to
    personally retrieve the fax as it is being sent.

-   **E-mail** may **NOT** be used for consensus materials unless the
    materials are purely administrative (e.g., agendas, ground rules for
    the calls), and they contain no mention of the applicant.

-   **Phones** must be secure---cellular and portable phones are **NOT**
    secure.

**[Problems:]{.underline}** If your schedule has changed and you are no
longer available to participate in consensus, or if you have any
concerns or questions, please contact:

Debbie Smyth 301-975-5312 Bob Fangmeyer 301-975-4781

Sandra Byrne 301-975-4812 Mike Berry 301-975-8212

Mark Shapiro 301-975-3621

** [BACK-UP TEAM LEADER---ORIGINAL READ]{.underline}**

\_\_\_\_\_Your team consists of Examiners who have all read and scored
the application in Stage 1.

\_\_\_\_\_One or more members of your team are new to the application in
Stage 2: Consensus Review. New team members have until **August 17,
2004,** to return their completed Key Themes Worksheet to ASQ. ASQ will
then forward the Key Themes Worksheets to you and all team members.

ENCLOSED:

  --- ------------------------------- ---- ---------------------------------
  1   Urgent Reply Form **(Fax,       7    Stage 1: Independent Review
      phone, or e-mail ASQ                 scorebooks
      immediately receipt of               
      package)**                           

  2   Updated Dates Unavailable Forms 8    Scoring data (*Stage 1 Examiner
      (**submit to team leader by          Scoring Table, Stage 1 Scoring
      August 6, 2004)**                    Graph, Stage 1 Frequency Table)*

  3   Blank Biography Form            9    Scoring Guidelines with Comment
      **(complete and send to team         Guidelines on reverse side
      leader by August 6, 2004)**          

  4   Team Roster                     10   Peer Evaluation Forms and
                                           envelopes to return to NIST

  5   Quickstart Instructions for     11   Federal Express package to return
      Team Leaders                         final consensus scorebook to NIST
                                           by September 14, 2004 **(In the
                                           event that the backup assumes the
                                           team leader role)**

  6   Consensus Manual                     
  --- ------------------------------- ---- ---------------------------------

** [BACKUP TEAM LEADER---NEW READ]{.underline}**

As a new read to this team, **you must complete and return your Key
Themes Worksheet for the enclosed application to ASQ no later than
August 17, 2004, via fax at (414) 765-7214**.

\_\_\_\_\_Your team consists of Examiners who have all read and scored
the application in Stage 1.

\_\_\_\_\_One or more members of your team is new to the application in
Stage 2: Consensus Review. You will receive a copy of any new team
member's Key Themes Worksheet on or before August 19, 2004.

ENCLOSED:

  --- ----------------------------- ---- ----------------------------------
  1   Urgent Fax Form **(Fax        7    Consensus Manual
      immediately to ASQ to confirm      
      receipt of package)**              

  2   Application                   8    Stage 1: Independent Review
                                         scorebooks (*except for new team
                                         member\[s\])*

  3   Updated Dates Unavailable     9    Scoring data (*Stage 1 Examiner
      Forms **(submit to team            Scoring Table, Stage 1 Scoring
      leader by August 6, 2004)**        Graph, Stage 1 Frequency Table)*

  4   Blank Biography Form          10   Scoring Guidelines with Comment
      **(complete and send to team       Guidelines on reverse side
      leader by August 6, 2004)**        

  5   Team Roster                   11   Federal Express package to return
                                         final consensus scorebook to NIST
                                         by September 14, 2004 **(In the
                                         event that the backup assumes the
                                         team leader role)**

  6   Quickstart Instructions for   12   Procedures and Federal Express
      Team Leaders                       envelope/label for returning
                                         materials to ASQ at the completion
                                         of Award Cycle
  --- ----------------------------- ---- ----------------------------------

Again, thank you for the commitment of your time during the 2004 Award
cycle.

Peter LaBonte, MBNQA Program

Enclosures

# LETTER TO TEAM MEMBERS

Dear *Examiner*:

Thank you for being part of the Consensus Team for #0XX-00.

**[Enclosures:]{.underline}** The instructions and materials will vary
depending on whether you are an original read to the team (you read the
application at Stage 1) or you are a new read to the team (you did not
read the application at Stage 1). On the page that follows, please find
the check box that corresponds to your situation, and ensure that you
have the materials listed under that check box. Please call the ASQ
Baldrige Department immediately at 414-765-7205 if you are missing any
materials.

***Note: [If you are a new read to this team,]{.underline} you must read
and evaluate the application and FAX a completed Key Themes Worksheet to
ASQ at 414-765-7214 no later than August** **17,** **2004.** Upon
receipt of your Key Themes Worksheet, ASQ will send you the remainder of
the team's evaluation and scoring material. ASQ will then forward your
Key Themes Worksheet to each team member.*

**[Team Leader Contact:]{.underline}** The team leader will contact you
by August 4, 2004, to discuss the consensus schedule and tentative team
assignments. Remember that the team leader is trying to coordinate the
schedule of six team members, so please be as flexible as possible.
**The team leader will also ask you to submit a biography form and
updated Dates Unavailable Forms by August 6.**

**[Getting Started:]{.underline}** Preparation is a critical aspect of
the consensus process; please review the enclosed Consensus Manual. It
is also available on-line at
[www.baldrige.nist.gov/Examiner_Resources.htm](http://www.baldrige.nist.gov/Examiner_Resources.htm).
Please pay particular attention to the initial steps in the planning
process, e.g., scheduling calls, preparing instructions for the team.
Planning calls must occur between August 11 and August 18; consensus
calls must be held August 30 through September 3 and September 7 and 8.

**[Protecting confidentiality]{.underline}** of the applicant is
paramount and a challenge during the consensus process.

-   In your **written work or in discussions** with your team members,
    refer to the applicant only by its number, as "the organization," or
    as "the applicant."

-   When **faxing** materials to your team members, the receiving fax
    machine must be secure or the recipient must be available to
    personally retrieve the fax as it is being sent.

-   **E-mail** may **NOT** be used for consensus materials unless the
    materials are purely administrative (e.g., agendas, ground rules for
    the calls), and they contain no mention of the applicant.

-   **Phones** must be secure---cellular and portable phones are **NOT**
    secure.

**[Problems:]{.underline}** If your schedule has changed and you are no
longer available to participate in consensus, or if you have any
concerns or questions, please contact:

Debbie Smyth 301-975-5312 Bob Fangmeyer 301-975-4781

Sandra Byrne 301-975-4812 Mike Berry 301-975-8212

Mark Shapiro 301-975-3621

** [TEAM MEMBER---ORIGINAL READ]{.underline}**

\_\_\_\_\_ Your team consists of Examiners who have all read and scored
the application in Stage 1.

\_\_\_\_\_ One or more members of your team are new to the application
in Stage 2: Consensus Review. You will receive a copy of any new team
member's Key Themes Worksheet on or before August 19, 2004.

ENCLOSED:

  --- ------------------------------- --- ----------------------------------
  1   Urgent Reply Form **(Fax,       5   Consensus Manual
      phone, or e-mail ASQ                
      immediately receipt of              
      package)**                          

  2   Updated Dates Unavailable Forms 6   Stage 1: Independent Review
      **(submit to team leader by         scorebooks
      August 6, 2004)**                   

  3   Blank Biography Form            7   Scoring data (*Stage 1 Examiner
      **(complete and send to team        Scoring Table, Stage 1 Scoring
      leader by August 6, 2004)**         Graph, Stage 1 Frequency Table)*

  4   Team Roster                     8   Scoring Guidelines with Comment
                                          Guidelines on reverse side
  --- ------------------------------- --- ----------------------------------

** [TEAM MEMBER---NEW READ]{.underline}**

As a new read to this team, y**ou must complete and return your Key
Themes Worksheet for the enclosed application to ASQ no later than
August 17, 2004, via fax at (414) 765-7214**. After you and any other
new team members have returned your completed Key Themes Worksheets to
ASQ, it will send you a roster of team members, copies of all Stage 1
scorebooks and additional Key Themes, and the summary scoring data.

ENCLOSED:

  --- ------------------------------- --- ----------------------------------
  1   Urgent Reply Form **(Fax,       5   Consensus Manual
      phone, or e-mail ASQ                
      immediately receipt of              
      package)**                          

  2   Application                     6   Scoring Guidelines with Comment
                                          Guidelines on reverse side

  3   Updated Dates Unavailable Forms 7   Procedures and Federal Express
      **(submit to team leader by         envelope/label for returning
      August 6, 2004)**                   materials to ASQ at the completion
                                          of Award cycle

  4   Blank Biography Form                
      **(complete and send to team        
      members by August 6, 2004)**        
  --- ------------------------------- --- ----------------------------------

Thank you for the commitment of your time during the 2004 Award cycle.

Peter LaBonte, MBNQA Program

Enclosures

***Section Five***

***Communication Among Team Members***

**TEAM MEMBERS TO TEAM LEADER**

**Dates Unavailable Form: Planning Call**

**Fax to Team Leader by *[August 6 ]{.underline}***

To:\_\_\_\_\_\_\_\_\_\_\_\_\_\_(Team Leader) From:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_(Examiner)

I am **[unavailable]{.underline}** to participate in conference calls
for the consensus process at the following times on the specified days
below. The times are indicated in **[Eastern Daylight Time
(EDT)]{.underline}**.

+---------+------+------+------+------+------+------+------+------+
| *       | **8/ | **8/ | **8/ | **8/ | **8/ | **8/ | **8/ | **8/ |
| *Time** | 11** | 12** | 13** | 14** | 15** | 16** | 17** | 18** |
|         |      |      |      |      |      |      |      |      |
| **EDT** | **W  | *    | **F  | **S  | **S  | **M  | **Tu | **W  |
|         | ed** | *Thu | ri** | at** | un** | on** | es** | ed** |
|         |      | rs** |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **6:00  |      |      |      |      |      |      |      |      |
| AM**    |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *6:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *7:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *7:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *8:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *8:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *9:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *9:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 10:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 10:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 11:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 11:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *Noon** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **12:30 |      |      |      |      |      |      |      |      |
| PM**    |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *1:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *1:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *2:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *2:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *3:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *3:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *4:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *4:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *5:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *5:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *6:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *6:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *7:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *7:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *8:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *8:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *9:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| *       |      |      |      |      |      |      |      |      |
| *9:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 10:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 10:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 11:00** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **      |      |      |      |      |      |      |      |      |
| 11:30** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+
| **Mid   |      |      |      |      |      |      |      |      |
| night** |      |      |      |      |      |      |      |      |
+---------+------+------+------+------+------+------+------+------+

**Dates Unavailable Form: Consensus Calls**

**Fax to Team Leader by *[August 6 ]{.underline}***

To:\_\_\_\_\_\_\_\_\_\_\_\_\_\_(Team Leader) From:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_(Examiner)

I am **[unavailable]{.underline}** to participate in conference calls
for the consensus process at the following times on the specified days
below. The times are indicated in **[Eastern Daylight Time
(EDT)]{.underline}**.

+----------+-------+-------+-------+-------+-------+-------+-------+
| **Time** | **8   | **8   | **    | **    | **    | **    | **    |
|          | /30** | /31** | 9/1** | 9/2** | 9/3** | 9/7** | 9/8** |
| **EDT**  |       |       |       |       |       |       |       |
|          | **    | **T   | **    | **Th  | **    | **T   | **    |
|          | Mon** | ues** | Wed** | urs** | Fri** | ues** | Wed** |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **6:00   |       |       |       |       |       |       |       |
| AM**     |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **6:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **7:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **7:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **8:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **8:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **9:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **9:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *10:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *10:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *11:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *11:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **Noon** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **12:30  |       |       |       |       |       |       |       |
| PM**     |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **1:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **1:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **2:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **2:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **3:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **3:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **4:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **4:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **5:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **5:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **6:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **6:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **7:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **7:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **8:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **8:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **9:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **9:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *10:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *10:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *11:00** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| *        |       |       |       |       |       |       |       |
| *11:30** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+
| **Mi     |       |       |       |       |       |       |       |
| dnight** |       |       |       |       |       |       |       |
+----------+-------+-------+-------+-------+-------+-------+-------+

**Biography Form**

**Please complete and fax this form to the team leader by** ***August
6***.

TO: Team Leader

FROM:

Subject: Biography

***Current Employer---short description of current
job/responsibilities:***

***Experience in the Baldrige Review Process:***

***Work Experience in the Baldrige Categories:***

***Background/experience relevant to the applicant's business:***

***On a personal note (optional):***

**Sample Completed Biography Form**

TO: Team Leader - Consensus Team #XX

####### FROM: Jane Doe

Subject: Biography

***Current employer---short description of job/responsibilities:***

I currently work for XYZ corporation as the Director of Strategic
Planning. I manage a staff of 10 people who oversee the strategic
planning process for our division. We assist department managers in
identifying future needs and materials that align with the strategic
objectives set by our parent organization.

***Experience in the Baldrige Review Process:***

This is my third year as a Malcolm Baldrige Examiner. I am also a Judge
in my state award program. At the national level, I have participated in
the consensus and site visit processes (for a health care applicant). As
a Judge at the state level, I have reviewed and recommended state award
recipients from business and health care organizations.

National Level:

Led Category 2 -- Strategic Planning

Backup for Customer and Market Focus

My strongest categories are:

Category 2 -- Strategic Planning

Category 4 -- Measurement, Analysis, and Knowledge Management

My weakest categories are:

Category 3 -- Customer and Market Focus

Category 5 -- Human Resource Focus

***Work Experience in the Baldrige Categories:***

Extensive experience in Categories 1, 2, and 7 within my organization
and formerly as a consultant in my own business.

***Background/experience relevant to the applicant's business:***

Minimal experience directly related to the specific business, but I have
a lot of experience from prior work with service organizations that have
over 700 employees.

***On a personal note:***

My husband and I are active in our church and community. We enjoy
traveling and working in the yard.

**TEAM LEADER TO TEAM MEMBERS**

**SAMPLE Team Leader Planning Letter**

*Date*

####### To: Consensus Review Team Members - Applicant XXX

From: (Team Leader name)

Welcome to Team *YR-XXX*! Having spoken with each of you, I am happy to
report that NIST has assembled a team with great experience and
diversity. I look forward to working with you and doing an excellent job
for our applicant and the Judges. We have a lot to do!

**Consensus Review Team Calls**: to reconfirm our schedule, we have set
up the following calls.

  ------------------- ------------------------- -------------------------
  **Type of Call**    **Date**                  **Time**

  Planning Call       Wednesday, August 18      2:00--3:00 p.m. EDT

  Consensus Call      Wednesday, September 1    9:00 a.m.--1:00 p.m. EDT

  Consensus Call      Thursday, September 2     12:00 noon--4:00 p.m. EDT
  ------------------- ------------------------- -------------------------

You should all be receiving call instructions from ASQ shortly.

**Item and Other Assignments:**

Thank you for providing your biography with Item preferences. I have
tried to balance your preferences, experience, and industry knowledge
with team needs. Here are the assignments:

+----------+-------------+-------------+-----------------+------------+
| **Ex     | **Ca        | **Backup**  | **Other**       | *          |
| aminer** | tegory/Item |             |                 | *Scorebook |
|          | Lead**      |             |                 | Sponsor**  |
+----------+-------------+-------------+-----------------+------------+
| Team     | Items 1.1,  | Items 2.1,  |                 |            |
| Leader   | 1.2, 7.6    | 2.2, 7.3    |                 |            |
| #1       |             |             |                 |            |
+----------+-------------+-------------+-----------------+------------+
| Backup   | #####       | Items 5.1,  | Scorebook       |            |
| Team     | ## Items 3. | 5.2, 5.3,   | Editor, Key     |            |
| Leader   | 1, 3.2, 7.1 | 7.4         | Themes          |            |
| #2       |             |             | Worksheet       |            |
+----------+-------------+-------------+-----------------+------------+
| Examiner | Items 2.1,  | Items 1.1,  | Tim             | Examiner   |
| #3       | 2.2, 7.3    | 1.2, 7.6    | ekeeper/process | #1222      |
|          |             |             | checker         |            |
+----------+-------------+-------------+-----------------+------------+
| Examiner | Items 6.1,  | Items 4.1,  | Scorekeeper,    |            |
| #4       | 6.2, 7.5    | 4.2, 7.2    | Score Summary   |            |
|          |             |             | Worksheet       |            |
+----------+-------------+-------------+-----------------+------------+
| Examiner | Items 5.1,  | Items 3.1,  | Computer expert |            |
| #5       | 5.2, 5.3,   | 3.2, 7.1    |                 |            |
|          | 7.4         |             |                 |            |
+----------+-------------+-------------+-----------------+------------+
| Examiner | Items 4.1,  | Items 6.1,  | Key Factors     | Examiner   |
| #6       | 4.2, 7.2    | 6.2, 7.5    | Worksheet,      | #655       |
|          |             |             | Criteria cop    |            |
+----------+-------------+-------------+-----------------+------------+

**[Computer Use]{.underline}**

Are we all PC compatible? Can we all save/read CDs?

Use NIST electronic scorebook from Web at
http://www.quality.nist.gov/04scorebook.htm, (WORD 6.0/95, Times New
Roman, 12 point font)

Save on diskettes or CDs, not on hard drive.

Establish saving protocol, (e.g. save under Item number and version
(e.g., Item 3.1 draft 1, 2, 3, etc.).

Use diskette/CD mailer to send to scorebook editor.

Close document before leaving your desk.

Do not destroy files until you have been instructed to do so by ASQ.

Use e-mail only for logistics, not for applicant-specific information.

Return application to ASQ when instructed to do so.

**[Speakerphones and Headsets]{.underline}**

I recommend the use of a headset for all of our calls. While it may be
easier to use a speakerphone, poor sound quality and noise from handling
papers often interfere with the conversation. This is especially true if
the speakerphone is sound activated. If you do use a speakerphone, use
the mute button until you are ready to talk. You may wish to try out
sound quality prior to the calls. Headsets have many advantages and
often can be borrowed from your company. Headsets also can be purchased
for about \$45.00 from electronics stores.

**[Call In Time]{.underline}**

I have found it is a good idea to dial in for the conference call five
minutes early so you do not run into a delay getting connected by the
operator. When team members are delayed in joining the calls, it
requires adjustments in the schedule that could be difficult for the
team. Your promptness will be appreciated by all.

**[Planning Call Agenda]{.underline}**

I have attached a proposed agenda for our planning call. During the
call, we will begin to get to know each other, and we will clarify our
work process. In preparation for that call, the scorebook editor and I
will fax to you proposed Key Factors and the draft Key Themes
Worksheets.

**[Proposed Ground Rules]{.underline}**

Here are some proposed ground rules. We will discuss these before the
consensus calls, and we can modify them as we see fit.

-   Be on time for all calls. Call in early to be sure you are connected
    at the start of the call.

```{=html}
<!-- -->
```
-   Identify yourself when speaking. For example, "This is XXXXXX. I
    suggest\...."

```{=html}
<!-- -->
```
-   Focus scoring discussions using the applicable Scoring Guidelines.

-   No one "owns" any Item. We are trying to get the best collective
    thinking.

-   We begin and end on time by sticking to the overall schedule of
    events. We listen to our timekeeper. We actively manage our time.

-   No **e-mail; only secure fax, diskette, or telephone for information
    transfer regarding the applicant.**

-   Reports and paperwork are completed on time. We honor our scorebook
    editor's requests. We ask for help as needed to meet obligations.

-   We will commit ourselves to utilizing Stage 1 scorebooks of
    Examiners not participating in the consensus process.

-   As we do our work, we will make every effort to draft consolidated
    feedback-ready comments.

-   We listen to other points of views, particularly those that are
    different, in addition to expressing our own. We can and should
    offer different perspectives, but we are striving for consensus.
    Consensus means we have heard, have been heard, and thus can and
    will support the team's outcomes.

-   We will not proceed with the consensus call if any team member is
    missing.

-   Each team member is responsible for the success of the team.

-   HAVE ***FUN!***

-   Others?

**[REMINDERS]{.underline}**

XXXXX will be the NIST monitor for our calls. While an observer to the
team, the NIST monitor plays an active role reviewing our process. So,
please send the NIST monitor a copy of any materials that you send to
everyone on the team.

Please review the just-in-time materials at
www.baldrige.nist.gov/Examiner_Resources.htm.

Please remember, **do not fax** any material that in any way identifies
the applicant until you have determined the fax is secure or that the
person is available to receive it. If you are unable to fax securely,
use Federal Express. You may use ASQ's Federal Express number that
appears on the Federal Express airbills.

I also want to make a special request that each Examiner make every
effort to draft feedback-ready comments. This will make compiling the
final consensus scorebook more efficient. Also, our consensus
discussions will be more productive.

As you prepare your work, you are to review and use the input from all
scorebooks prepared at Stage 1, including those of Examiners who are not
participating in the consensus process.

Please do not hesitate to call me in my office or at home for any
reason. If I am traveling, I check my voice mail several times a day, so
I should be able to respond quickly to any needs. I am looking forward
to working and talking with each of you.

**SAMPLE PLANNING CALL AGENDA**

**[Planning Call---Application #XXX]{.underline}**

**[Proposed Agenda]{.underline}**

***Date and Time of Call***

5 minutes Introduction (team and NIST/BNQP monitor)

1 minute Reconnect procedures

2 minutes Confirm materials received and no cellular phones being used

4 minutes Ground rules---review proposed rules, suggest additions or

> modifications

2 minutes Computer support---determine process and format, address
issues

8 minutes Review proposed Key Factors and Key Themes Worksheet if new
members on the team have submitted their completed Key Themes to ASQ by
the time of the call

10 minutes Review team members' Item and other assignments

10 minutes Discuss/agree on preparation required for consensus review
process

5 minutes Discuss the role of the scorebook editor

8 minutes Review/approve consensus call agendas

2 minutes Review timeline from now to submission of consensus scorebook

2 minutes Review/update address, phone, fax, e-mail (note secure or
not); provide

> confidentiality reminder

2 minutes Check site visit availability

2 minutes Closing---reminder: peer evaluation forms

**SAMPLE**

**Consensus Call Process**

*Date*

TO: Consensus Team Members

From: Team Leader

Greetings. Thank you for your responsiveness regarding the start-up of
our work together. I have outlined process requirements for preparing us
for the consensus call.

**[Homework and Preparation for Consensus Call]{.underline}**

Following are proposed instructions to get ready for the consensus call:

-   Using all of the assigned scorebooks, including those of Stage 1
    Examiners not participating in consensus prepare a draft Item
    worksheet for each Item on which you are the lead. The draft should
    contain 4--6 key factors for the Item and a summary of key (6--10)
    strengths and opportunities for improvement. Your summary should be
    a synthesis of the collective thinking of the team.

-   Reread the Comment Guidelines on the reverse side of the applicable
    Scoring Guidelines. Follow the Comment Guidelines. For example, in
    most cases you will have 6--10 comments per Item; each comment will
    contain a single, complete thought. By using no more than 10
    comments, you will be identifying only the most important points for
    each Item. Also use the *Guidance for Producing a Well-Written
    Scorebook* contained in the scorebook instructions to assist you in
    comment writing.

-   Identify issues causing scoring differences and the reasons.

-   Indicate the scoring range you recommend (e.g., 30--45%). Have a
    specific score in mind, but wait until after the team reaches
    consensus on the scoring range before proposing the score. This
    keeps your mind more open to accept and integrate input from the
    consensus discussion.

-   Refer to the applicant only as "applicant" and not by the
    organization name to ensure confidentiality.

-   (Optional) Identify and list the top 2--4 SVIs per Item. These
    should be the most important areas to clarify or verify. Think
    through how you will pursue resolution of these issues, in the event
    the applicant is selected for a site visit. The reason for limiting
    the number of SVIs is that it is very difficult to address more than
    4, and the purpose is to verify/clariy only **key** issues.

-   Send your draft Item Worksheets to your backup. Use a secure fax or
    Federal Express. The role of the Category/Item backup is to give
    constructive feedback to the Category/Item lead, focusing on the
    central requirements of the Criteria, in light of the key factors
    for this applicant. This process should be
    **[completed]{.underline}** by *XXXXX,* at the latest. Earlier is
    better.

####### Revise your draft Item worksheets, using input from your backup, and fax (or Federal Express) a

copy of the final draft of your Item Worksheets to all teammates and to
the NIST monitor to arrive by *XXXXX*. When faxing, remember to call
before sending if the recipient does not have a secure fax, to maintain
confidentiality.

-   Read all the draft consensus materials from your teammates in
    preparation for our consensus call on *XXXX*. Review Item Worksheets
    to identify any issues raised in the Stage 1 scorebook for which you
    are the sponsor that are not covered in the drafts.

**[Consensus Call]{.underline}**

By this time, there will have been quite a bit of Examiner interaction
through dialogue between Category/Item leads and backups. The purpose of
the call is to reach consensus among all of the team members on all
parts of the consensus scorebook.

#### Proposed process for Item discussion

The lead for each Item presents

-   a brief summary of the Criteria requirements

-   key factors relevant to the Items; note agreement with or
    differences from those on the draft Key Factors Worksheet

-   strengths/OFIs; note agreement/differences

-   summary of comment differences causing scoring differences

-   proposed scoring range

-   site visit issues (if applicable)

The team discusses

-   the most important strengths and OFIs and resolves differences in
    team members' evaluation of the applicant

-   input, as appropriate, from scorebooks of Stage 1 Examiners not on
    the call

-   the scoring range

-   the consensus score

```{=html}
<!-- -->
```
-   site visit issues (if applicable)

The lead for the Item

-   summarizes and records the discussion

-   checks for consensus on comments, the scoring range, and the score
    (and site visit issues, if applicable)

**[Post-Consensus Call Requirements]{.underline}**

Please complete the following steps after the consensus call.

+-----------------------+----------------------+----------------------+
| #### WHAT             | #### WHO             | #### WHEN COMPLETED  |
+-----------------------+----------------------+----------------------+
| Call consensus        | Team Leader          | Immediately after    |
| hotline               |                      | each consensus call  |
| (1-800-898-4506) to   |                      |                      |
| report that call is   |                      |                      |
| finished              |                      |                      |
+-----------------------+----------------------+----------------------+
| Send Score Summary    | Team Leader          | Upon completion of   |
| Worksheet to ASQ      |                      | final consensus call |
+-----------------------+----------------------+----------------------+
| Revise Item           | All                  | *Date*               |
| Worksheets per        |                      |                      |
| decisions made on the |                      |                      |
| consensus call        |                      |                      |
+-----------------------+----------------------+----------------------+
| Complete and send     | All                  | *Date*               |
| Peer Evaluation Forms |                      |                      |
| to NIST               |                      |                      |
+-----------------------+----------------------+----------------------+
| Send paper copy of    | ####### All          | For arrival no later |
| revised Item          |                      | than *Date*          |
| Worksheets via        |                      |                      |
| Federal Express to    |                      |                      |
| team leader,          |                      |                      |
| scorebook editor and  |                      |                      |
| NIST monitor; in      |                      |                      |
| addition, send copy   |                      |                      |
| on diskette to        |                      |                      |
| scorebook editor      |                      |                      |
+-----------------------+----------------------+----------------------+
| Draft the consensus   | Scorebook Editor     | *Date*               |
| scorebook             |                      |                      |
+-----------------------+----------------------+----------------------+
| Send final scorebook  | Scorebook Editor/    | For arrival no later |
| to NIST               |                      | than *Date*          |
|                       | Team Leader          |                      |
+-----------------------+----------------------+----------------------+

Thanks for reviewing all of this material. Your hard work is
appreciated, and I look forward to working

with each of you!

cc: NIST/BNQP Monitor

**SAMPLE**

**Consensus Review Schedule**

**(for intact team)**

**APPLICANT XX-XXX**

MO/DAY Fax biography and updated Dates Unavailable Forms to team leader.

MO/DAY Conduct the consensus planning call.

MO/DAY Exchange draft Item Worksheets with Category/Item backup.

MO/DAY Distribute the second draft of Item Worksheets and all other
drafts to team members and the NIST monitor.

MO/DAY Conduct the consensus call.

MO/DAY Conduct the consensus call.

MO/DAY Send revised Item Worksheets and all other worksheets to the team
leader, scorebook editor, and NIST monitor.

MO/DAY Send scorebook peer reviews to NIST for distribution.

MO/DAY Scorebook editor sends consensus scorebook to the team leader for
review.

MO/DAY Team leader approves the final consensus scorebook.

MO/DAY Team leader sends to NIST by Federal Express the final consensus
scorebook to NIST with feedback-ready comments (paper copy and
diskette/CD for all word

processed portions) for arrival no later than ***September 14.***

**SAMPLE**

**Consensus Call AgendaS**

*Dates and Times of Calls*

TO: Consensus Team Members

From: (Team Leader - Name)

Cc: NIST Monitor

Thanks again for the time spent and input provided during the planning
call the other day. I hope you all felt it was as productive as I did.
The time spent, and the agreements made, should help the consensus calls
go very smoothly and efficiently.

#### 

We will use the following agenda for the consensus calls:

#### Day 1

Introductions, Ground Rules,

Reconnection/break procedures Team Leader 5 minutes

Review agenda and roles Team leader 3 minutes

Review draft key factors Examiner #6 5 minutes

Review draft Key Themes Worksheet Scorebook editor 10 minutes

Items 1.1, 1.2, 7.6 Team leader 40 minutes

####### Process check Process checker 10 minutes

(NIST monitor may leave the call after the check)

Items 3.1, 3.2, 7.2 Examiner #2 40 minutes

####### Items 6.1, 6.2, , 7.5 Examiner #4 40 minutes

####### Recap Scores Scorekeeper 5 minutes

####### Call consensus hotline (1-800-898-4506) Team leader 1 minute

#### Day 2

Introduction/recap/questions for NIST monitor 10 minutes

Items 4.1, 4.2, 7.1 Examiner #6 40 minutes

Items 5.1, 5.2, 5.3, 7.4 Examiner #5 50 minutes

Items 2.1, 2.2, 7.3 Examiner #3 40 minutes

Confirm scores Scorekeeper 5 minutes

Finalize key factors Examiner #6 5 minutes

Finalize key themes Scorebook editor 15 minutes

Next Steps Team leader 10 minutes

Call consensus hotline (1-800-898-4506) Team leader 1 minute

**SAMPLE Team Leader Thank You Note**

*Date*

Dear Consensus Review Team Members:

I would like to express my appreciation to each of you for your personal
effort and teamwork in preparing for and reaching consensus on
application \# -XXX.

Your prework, leadership, teamwork, and follow-up made my task much
easier than it might have been. As a team, we have provided the
applicant with a fair and thorough review, and we have given the Judges
the information they will need to make their determination.

One final request: if you have not done so already, please complete the
Peer Evaluation Forms and return them to NIST/BNQP. You were provided
the forms and an envelope addressed to NIST in an earlier mailing from
ASQ.

Once again, thank you for your assistance.

Sincerely,

Team Leader

***Section Six***

***Instructions, Samples, and Tools for the Consensus Process***

######   {#section-15 .unnumbered}

###### Steps for Creating a Draft Consensus Item Worksheet

# 

1.  Isolate your Items from the rest of the Stage 1 scorebooks.

-   Physically separate your Items from the rest of the scorebooks to
    minimize confusion and to help you organize your work.

2.  Begin by reviewing the Criteria requirements.

3.  Select 4--6 key factors.

4.  Using all Stage 1 scorebooks, identify the core idea of each
    comment, noting similarities and differences in Examiners'
    evaluations of the applicant.

-   Be sure to include the ideas from "phantom" scorebooks, if any.

5.  Determine the 6--10 most important and relevant core ideas for the
    applicant.

6.  Write a comment for each of the 6--10 core ideas. Start with the
    best statement of the ideas in Stage 1 and add from others as
    needed. (For an example, see "Synthesizing a Stage 2 Comment from
    Four Stage 1 Comments, p. XX)

-   Note in parentheses after the comment which Examines had the same
    core ideas in their Stage 1 scorebooks.

```{=html}
<!-- -->
```
-   Include outlier comments if they rise to the level of the 6--10 core
    ideas for the applicant.

7.  Following your 6--10 comments, insert a section with Stage 1
    scorebook comments that were not used (outliers and other comments),
    and include the reason these comments were ommitted (e.g., beyond
    the criteria)

8.  Propose a solution to resolve any conflicts between strengths and
    OFIs.

-   Not every Examiner will agree during the independent evaluation ---
    one Examiner might see an approach as a strength, while another may
    see it as an OFI. You must propose a resolution as part of your
    drafting task.

9.  Finalize your comments using the Comment Guidelines.

-   It is important to make draft comments "feedback ready."

```{=html}
<!-- -->
```
-   Propose a scoring range.

10. If directed by your team leader, draft 2--4 site visit issues based
    on the comments.

11. Send draft comments to the Item backup.

12. Revise drafts, incorporating the backup's input.

Synthesizing a Stage 2 Comment from Four Stage 1 Comments

Comments from four different Stage 1 scorebooks on the same area to
address:

+---+------+----+------------------------------------------------------+
| + | Item | KF |                                                      |
| / |      |    |                                                      |
| + | Ref. | Re |                                                      |
| + |      | f. |                                                      |
+---+------+----+------------------------------------------------------+
| \ | 1.1  |    | The Steering Team meets weekly to review             |
| + | c1,3 |    | organizational capabilities and performance          |
|   |      |    | regarding associate activities; supplier and partner |
|   |      |    | technology capabilities; safety, health,             |
|   |      |    | environment, and community relations; and progress   |
|   |      |    | on strategy. **Monthly Navigation Reviews are used   |
|   |      |    | to identify improvements and areas where strategy is |
|   |      |    | at risk. Kaizen improvement project teams are        |
|   |      |    | chartered and deployed throughout the organization   |
|   |      |    | to address at-risk areas.**                          |
+---+------+----+------------------------------------------------------+
| \ | 1.   |    | The Steering Team made up of the senior leaders      |
| + | 1c.1 |    | meets weekly to review and discuss the applicant's   |
|   |      |    | organizational performance and capabilities. Each    |
|   |      |    | meeting is convened around one of four topic areas:  |
|   |      |    | *(1) Successful Associates/Successful Teams; (2)     |
|   |      |    | Right Technology; (3) Right Environment; and (4)     |
|   |      |    | Navigation Reviews (progress to the Strategy Map).*  |
+---+------+----+------------------------------------------------------+
| \ | 1.   |    | KAIZEN Improvement Project Teams are formed by the   |
| + | 1c.3 |    | Steering Team when performance reviews indicate that |
|   |      |    | a performance target is at risk. KAIZEN teams are    |
|   |      |    | directed at the level where the target is at risk,   |
|   |      |    | either at the department/team level or sitewide.     |
|   |      |    | [Suppliers and partners are involved in KAIZEN teams |
|   |      |    | when their performance is a contributing             |
|   |      |    | factor.]{.underline}                                 |
+---+------+----+------------------------------------------------------+
| \ | 1.   |    | **[The Steering Team reviews performance weekly in   |
| + | 1c.1 |    | one of four topic areas including three of the five  |
|   |      |    | values]{.underline}**, and Navigation Reviews. The   |
|   |      |    | monthly Navigation Reviews focus on company-level    |
|   |      |    | measures, Course Coordinates and linked lower-level  |
|   |      |    | measures, to determine whether course corrections    |
|   |      |    | are needed to address changing organizational needs. |
+---+------+----+------------------------------------------------------+

> Sample of a synthesized Stage 2 comment on the same area to address:

+---+------+---+------------------------------------------------------+
| + | Item | K |                                                      |
| / |      | F |                                                      |
| + | Ref  |   |                                                      |
| + |      | R |                                                      |
|   |      | e |                                                      |
|   |      | f |                                                      |
+---+------+---+------------------------------------------------------+
| \ | 1.1c |   | **[The Steering Team reviews performance weekly in   |
| + | .1,3 |   | one of four topic areas (including three of the five |
|   |      |   | values): (1) Successful Associates/Successful        |
|   |      |   | Teams]{.underline}**; *(2) Right Technology; (3)     |
|   |      |   | Right Environment; and (4) Navigation Reviews        |
|   |      |   | (progress to the Strategy Map)*. The **monthly       |
|   |      |   | Navigation Reviews, which focus on company-level     |
|   |      |   | measures called Course Coordinates, are used to      |
|   |      |   | identify improvements and areas where strategy is at |
|   |      |   | risk. Kaizen improvement project teams are chartered |
|   |      |   | and deployed throughout the organization to address  |
|   |      |   | at-risk areas**. [Suppliers and partners are         |
|   |      |   | involved in KAIZEN teams when their performance is a |
|   |      |   | contributing factor.]{.underline}                    |
+---+------+---+------------------------------------------------------+

**Sample Script**

**Presentation of Item 1.1 Summary**

(based on attached Item 1.1 Worksheet)

**Criteria Requirements**---Item 1.1 asks how senior leaders set and
deploy values, short- and longer-term directions, and performance
expectations and balance the value for customers and other stakeholders
in performance expectations. This includes how leaders create an
environment for empowerment, innovation, organizational agility,
learning, and ethical behavior.

It also asks how the governance system ensures fiscal accountability and
independence in audits and how it protects stakeholder interest.

Item 1.1 asks how senior leaders review organizational performance, what
key performance measures are reviewed regularly, and how review findings
are used to drive improvement and innovation, including improvement in
the effectiveness of leaders and the board of directors.

## **Key Factors**

# 1. Mission---provide high-quality polymers that make customers more competitive and 

###  provide value to shareholders

2\. Vision of skilled associates developing and delivering plastics for
a healthy planet

3\. Three guiding principles

-   Support communities

# Achieve highest ethical standards 

-   Invest in a future society

4\. Five G-ORB Values

-   Growth through Partnerships

-   Right technology

-   Right cost

-   Right environment

```{=html}
<!-- -->
```
-   Successful associates/successful teams

5\. Board of Directors for the parent corporation composed of four
internal and eight external members; has

four standing committees: Nominating, Compensation, Audit, Litigation

6\. Performance improvement approaches include Hoshin Kanri and Kaizen
Improvement Process

## **Comment Summary**---I have five proposed strengths.

Comment 1. Focuses on the steering committee's use of the GSAC process
as part of the GPS to set and deploy short- and longer-term directions.

Comment 2. Focuses on how senior leaders communicate values, directions,
and expectations through the Hoshin Catchball Process, company
newsletter, and a number of other methods.

Comment 3. Focuses on the team-based culture, which facilitates target
setting and planning and contributes to an environment for empowerment
and associate involvement.

Comment 4. Focuses on the governance system, external and internal
audits, and how governance practices help ensure the protection of
stockholder and stakeholder interests.

Comment 5. Focuses on weekly organizational performance reviews, monthly
Navigation reviews, and the use of the Kaizen Improvement Project teams
to address at-risk areas.

**Strengths not used**: None

**Proposed OFIs---**I have three proposed OFIs.

Comment 1. It is unclear how the board's guiding principles or its
committee structure address management accountability, fiscal
accountability, or the protection of stakeholder interests at all levels
of the organization.

Comment 2. It is unclear how senior leaders translate performance review
findings into priorities for improvement and how both priorities and
opportunities for improvement are deployed throughout the organization
and suppliers and partners.

Comment 3. There is no evidence of a systematic process for evaluating
the performance of the CEO or the Board of Directors. It is unclear how
senior leaders use organizational performance review findings to improve
their leadership effectiveness.

**OFIs not included**

I did not include the OFI that focuses on how the applicant monitors
ethical behavior throughout the organization. This comment addresses
Item 1.2b, Ethical Behavior, not Item 1.1b, Organizational Governance.

**Proposed Site Visit Issues**---I propose four SVIs.

-   The first SVI covers Strengths 1 and 2 by verifying the Steering
    Team's use of the GSAC/GPS to establish and monitor short- and
    long-term direction and senior leadership's use of the Hoshin
    Catchball Process and other approaches to communicate values,
    directions, and expectations to associates.

-   The second SVI covers the governance system. It addresses Strength 4
    by verifying the approaches associated with the governance system,
    including performance and review of external audits, independent
    board members, and use of the Code of Conduct. It also covers OFI 1
    by clarifying how the governance system addresses management
    accountability, fiscal accountability, and the protection of
    stakeholder interests.

-   The third SVI covers Strength 5 by verifying that Gyroscope Course
    Coordinates are reviewed in monthly Navigation Reviews and that
    weekly reviews are focused on key topics, including values. It also
    covers OFI 2 by clarifying how senior leaders translate review
    findings into improvement priorities and deploy priorities through
    the organization and to suppliers/partners.

-   The fourth SVI addresses the refinement of the senior leadership
    system and improvement of senior leadership performance. This covers
    OFI 3 by verifying the use of the Leadership Assessment Session to
    evaluate Steering Committee performance and clarifying whether a
    systematic process exists for evaluating CEO/COD performance and how
    senior leaders use organizational performance review findings to
    improve the effectiveness of their own leadership or that of the
    leadership system.

```{=html}
<!-- -->
```
-   Strength 3 on the team-based culture is not covered by my SVIs. I
    recommend that it be covered by the Category 5 lead when work
    systems are verified, perhaps by walk-around questions.

**Score**

It appears that the applicant's approaches meet the overall requirements
of the Item but not the multiple requirements of the Item, especially in
the area of governance and organizational performance review. Deployment
varies in some areas or work units. I believe the applicant's "learning"
in terms of evaluation and improvement of the leadership system and
senior leadership performance is in the beginning stages, but the

learning from the Course Coordinates and the use of Kaizen improvement
teams appears to be effective. Overall, the best description of the
applicant's level of achievement in this Item is the 50--65 range.

I propose a scoring range of 50--65.

<table style="width:100%;">
<colgroup>
<col style="width: 5%" />
<col style="width: 0%" />
<col style="width: 5%" />
<col style="width: 0%" />
<col style="width: 5%" />
<col style="width: 0%" />
<col style="width: 6%" />
<col style="width: 0%" />
<col style="width: 24%" />
<col style="width: 47%" />
</colgroup>
<tbody>
<tr class="odd">
<td colspan="10">Item Worksheet—Item 1.1</td>
</tr>
<tr class="even">
<td colspan="10"><strong>Indicate the 4</strong>–<strong>6 most
important key business/organization factors relevant to this
Item.</strong></td>
</tr>
<tr class="odd">
<td colspan="9"><blockquote>
<p>1. Mission: Provide high-quality polymers that make customers more
competitive and provide value to shareholders.</p>
</blockquote>
<ol start="2" type="1">
<li><p>Vision is “Skilled associates developing and delivering plastics
for a healthy planet.”</p></li>
<li><p>Three guiding Principles: Support Communities, Achieve Highest
Ethical Standards, and Invest in a Future Society</p></li>
</ol>
<blockquote>
<p>4. Five Values: Growth Through Partnerships, Right Technology, Right
Cost, Right Environment, Successful Associates/Successful Teams</p>
</blockquote></td>
<td><blockquote>
<p><strong>5. Board of Directors for the parent corporation composed of
four internal and eight external, independent members; has four standing
committees: Nominating, Compensation, Audit, and Litigation</strong></p>
<p>6. Approaches to performance improvement include Hoshin Kanri for
business planning, the Kaizen Improvement Process, and knowledge sharing
through many mechanisms.</p>
</blockquote></td>
</tr>
<tr class="even">
<td colspan="10"><p><strong>Include an indication of the relative
importance/strength of the comment by using ++ or - - as
appropriate.</strong></p>
<p><strong>Include a reference to the most relevant Key
Factor(s).</strong></p></td>
</tr>
<tr class="odd">
<td><strong>+/++</strong></td>
<td colspan="2"><strong>Item Ref</strong></td>
<td colspan="3"><strong>KF Ref</strong></td>
<td colspan="2"><strong>ADLI</strong></td>
<td colspan="2"><strong>Strengths (Include figure references, as
appropriate.)</strong></td>
</tr>
<tr class="even">
<td colspan="10"><p>+ a1 1-3,6 A/I The applicant’s Steering Team, made
up of the President and his direct reports, is responsible for setting
and deploying short-term (one-year) and long-term (five-year) directions
that are aligned with the Vision, Principles, Mission, Goal, and Values
(Figure 1.1-1). The Steering Team uses the Gyroscope Semi-Annual
Calibration (GSAC) Process as part of the Gyroscope Planning System
(GPS) to establish, monitor, and revise organizational direction. (ALL,
NJ++)</p>
<p>+ a1 1-4 A Senior leaders communicate values, directions, and
expectations through a variety of approaches, including the Hoshin
Catchball Process, the <em>Compass</em> newsletter, GPS meetings,
recognition events, a Web page, closed-circuit television, banners and
posters, and new associate orientation. (BL, JJ, CC, JRL++)</p>
<p>+ a2 2,4,6 A/D The applicant’s team-based culture facilitates setting
performance targets and planning, assists associates in analysis and
problem solving, supports the development of new approaches and
innovative solutions, and creates an environment for empowerment and
associate involvement. Some teams operate at the work unit level, while
others are cross-functional or cross-product, and all associates are on
at least one team. (BL++, DJC, HF, JJ, NJ++, DWJ)</p>
<p>+ b 3, 5 A/D The applicant’s parent organization provides the primary
governance system. An external accounting firm conducts quarterly and
annual audits, and internal audits are conducted biannually. The Ethics
Committee, a standing committee under the Management Committee, reviews
all audit results. In addition, governance practices, including
appointment of independent board members and consistent use of the
<em>Code of Conduct,</em> help ensure protection of stockholder and
stakeholder interests. (DJC, FH,JJ,CC, NJ)</p>
<p>+ c1 1-4 A/D/I The Steering Team conducts weekly organizational
performance reviews in one of four topic areas that also address three
of the five Values. Monthly Navigation Reviews, which focus on
company-level measures called Course Coordinates, are used to identify
improvements and areas where strategy is at risk. Kaizen Improvement
Project Teams are chartered and deployed throughout the applicant’s
organization to address at-risk areas. (ALL)</p></td>
</tr>
<tr class="odd">
<td colspan="2"><strong>-/- -</strong></td>
<td colspan="2"><strong>Item Ref</strong></td>
<td><strong>KF Ref</strong></td>
<td colspan="2">ADLI</td>
<td colspan="3"><strong>Opportunities for Improvement (Include figure
references, as appropriate.)</strong></td>
</tr>
<tr class="even">
<td colspan="10"><p>- b 3,5 A/D Although the parent corporation’s Board
of Directors abides by principles for independent governance established
by the Japanese Corporate Governance Forum (JCGF), it is not clear how
these principles or the board’s committee structure addresses management
accountability, fiscal accountability, or the protection of stakeholder
interests at the level of the applicant’s organization. (DJC, JJ, DWJ,
JRL)</p>
<p>- c2,3 1-4 A/D The process by which senior leaders translate
performance review findings into priorities for improvement is not
evident, and recent review findings are not provided. In addition, it is
not clear how priorities and opportunities for improvement are deployed
throughout the applicant’s organization or to suppliers and partners to
ensure organizational alignment. (BL, DJC, JRL)</p>
<p>- c4 4 A/L Although the applicant uses an annual Leadership
Assessment Session to evaluate Steering Team performance, there is no
evidence of a process for systematically evaluating the performance of
the CEO or the Board of Directors. Also, it is not clear how senior
leaders use organizational performance review findings to improve the
effectiveness of their own leadership or that of the leadership system.
(BL, DJC, CC, DW- -)</p>
<p>OFI not included:</p>
<p>- b 3,5 A Other than audit results and participation in Partners in
Trust Seminars, it is not clear what measures or indicators the
applicant uses to monitor ethical behavior throughout the organization,
including among key partners, the Board of Directors, and other
stakeholders. Without such measures or indicators, the applicant may not
be able to fully assess performance levels for ethical behavior.
(JJ--)</p></td>
</tr>
<tr class="odd">
<td colspan="10"><p><strong>SITE VISIT ISSUES (For Stage 2, Consensus
Review, Only)</strong></p>
<ul>
<li><p>Verify Steering Team use of the GSAC/GPS to establish and
monitoring short and long term direction and senior leadership use of
the Hoshin Catchball Process and other approaches to communicate values,
directions, and expectations to associates. (Strength #1, 2)</p></li>
<li><p>Verify the approaches associated with the governance system,
including performance and review of external audits, independent board
members, and use of Code of Conduct. Clarify how the governance system
addresses management accountability, fiscal accountability, and the
protection of stakeholder interests. (Strength #4, OFI #1)</p></li>
<li><p>Verify Gyroscope Course Coordinates are reviewed in monthly
Navigation Reviews and that weekly reviews are focused on key topics and
value. Clarify how senior leaders translate review findings into
improvement priorities and deploy priorities through the organization
and to suppliers/partners. (Strength #5, OFI #2)</p></li>
<li><p>Verify the use of the Leadership Assessment Session to evaluate
Steering Committee performance. Clarify whether a systematic process
exists for evaluating CEO/COD performance and how senior leaders use
organizational performance review findings to improve the effectiveness
of their own leadership or that of the leadership system. (OFI
#3)</p></li>
</ul>
<p>Recommendation: Category 5 lead verify Strength #3 on team-based
culture and its deployment.</p></td>
</tr>
<tr class="even">
<td colspan="10"><p><strong>Change Due to Site Visit Findings (For Stage
3, Site Visit Review, Only)</strong></p>
<p><strong>     raise large      raise small      no change      lower
small      lower large</strong></p></td>
</tr>
</tbody>
</table>

**Item Worksheet---Item 1.1 Percent Score [50 - 60]{.underline}**

**Checklist for Process Checker to Use on Calls**

+-----+------+------+--------+-------+------+-----+------+---------+
| I   | Crit | Key  | St     | Areas | Sco  | Sc  | SVIs | Remarks |
| tem | eria | Fac  | rength | of    | ring | ore |      |         |
| \#  | Rq   | tors | s/OFIs | Dis   | R    |     | (If  |         |
|     | mts. |      |        | agree | ange | (Wh | app  |         |
| (In |      | (4   | (      | -ment |      | ole | lica |         |
| di  |      | --6) | 6--10) |       |      | num | ble) |         |
| scu |      |      |        |       |      | ber |      |         |
| ss- |      |      |        |       |      | %)  |      |         |
| ion |      |      |        |       |      |     |      |         |
| ord |      |      |        |       |      |     |      |         |
| er) |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+
|     |      |      |        |       |      |     |      |         |
+-----+------+------+--------+-------+------+-----+------+---------+

+--------------------+---+---+--------------------------------------------+---+---+
| Rules for          |   |   | Review the following rules for consensus   |   |   |
| Consensus Scoring  |   |   | scoring with the team scorekeeper:         |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   |  After an Item is discussed during the    |   |   |
|                    |   |   | consensus call(s), the assigned            |   |   |
|                    |   |   | Category/Item lead will propose a scoring  |   |   |
|                    |   |   | range. Team members will reach consensus   |   |   |
|                    |   |   | on the proposed scoring range. Once team   |   |   |
|                    |   |   | members reach consensus on the range, the  |   |   |
|                    |   |   | Item lead will propose a percent score     |   |   |
|                    |   |   | within the range. The team members then    |   |   |
|                    |   |   | will attempt to reach consensus on a       |   |   |
|                    |   |   | score.                                     |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   |  If, after a discussion, consensus cannot |   |   |
|                    |   |   | be reached and the difference in proposed  |   |   |
|                    |   |   | scores among team members participating on |   |   |
|                    |   |   | the call is 30% or less, the average of    |   |   |
|                    |   |   | the proposed scores of team members        |   |   |
|                    |   |   | participating in the conference call is    |   |   |
|                    |   |   | used.                                      |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | Review the following guidelines with the   |   |   |
|                    |   |   | team scorekeeper:                          |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ####### **Item Scores**                    |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | -   The consensus **percent** score for    |   |   |
|                    |   |   |     each Item should be a **whole** number |   |   |
|                    |   |   |     with no decimal places. To do this,    |   |   |
|                    |   |   |     use normal rounding rules (.5 and      |   |   |
|                    |   |   |     higher are rounded up; .49 and below   |   |   |
|                    |   |   |     are rounded down).                     |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ```{=html}                                 |   |   |
|                    |   |   | <!-- -->                                   |   |   |
|                    |   |   | ```                                        |   |   |
|                    |   |   | -   The consensus **points** for each Item |   |   |
|                    |   |   |     should be a **whole** number. To do    |   |   |
|                    |   |   |     this, multiply the consensus percent   |   |   |
|                    |   |   |     score (whole number) by the maximum    |   |   |
|                    |   |   |     points possible for the Item. Round to |   |   |
|                    |   |   |     the nearest whole number using normal  |   |   |
|                    |   |   |     rounding rules.                        |   |   |
+--------------------+---+---+--------------------------------------------+---+---+
|                    |   |   | ####### **Category Scores**                |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | -   The consensus **percent** score for    |   |   |
|                    |   |   |     each Category should be a **whole**    |   |   |
|                    |   |   |     number calculated by adding the point  |   |   |
|                    |   |   |     scores from all Items in the Category, |   |   |
|                    |   |   |     dividing the Category point score by   |   |   |
|                    |   |   |     the maximum points possible for the    |   |   |
|                    |   |   |     Category, and multiplying by 100 to    |   |   |
|                    |   |   |     convert to a percentage.               |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ####### **Grand Total**                    |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | -   The Grand Total of consensus           |   |   |
|                    |   |   |     **points** is arrived at by adding the |   |   |
|                    |   |   |     point scores for all seven Categories  |   |   |
|                    |   |   |     or all Items. It will be a whole       |   |   |
|                    |   |   |     number because all Item and Category   |   |   |
|                    |   |   |     point scores are whole numbers.        |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ```{=html}                                 |   |   |
|                    |   |   | <!-- -->                                   |   |   |
|                    |   |   | ```                                        |   |   |
|                    |   |   | -   The Grand Total consensus **percent**  |   |   |
|                    |   |   |     score is calculated by dividing the    |   |   |
|                    |   |   |     Grand Total consensus points by 10     |   |   |
|                    |   |   |     (example: 378 points = 37.8%) A        |   |   |
|                    |   |   |     decimal is okay here.                  |   |   |
+--------------------+---+---+--------------------------------------------+---+---+
| Tools Provided for |   |   | To help with the assignment of team        |   |   |
| Planning Consensus |   |   | members, the team leader will receive      |   |   |
| Assignments        |   |   | three documents:                           |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | 1.  Stage 1 Examiner Scoring Table         |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | 2.  Stage 1 Scoring Graph                  |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | 3.  Stage 1 Frequency of Scores            |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | A description of each document follows     |   |   |
|                    |   |   | with guidance on how it can be used in the |   |   |
|                    |   |   | consensus process. A sample copy of each   |   |   |
|                    |   |   | document is included.                      |   |   |
+--------------------+---+---+--------------------------------------------+---+---+
| Stage 1 Examiner   |   |   | The **Stage 1 Examiner Scoring Table** is  |   |   |
| Scoring Table      |   |   | a table showing the Item and total scores  |   |   |
|                    |   |   | for all Examiners who scored the           |   |   |
|                    |   |   | application at Stage 1, with summary data  |   |   |
|                    |   |   | for the application. The first two columns |   |   |
|                    |   |   | on the left show the Item number and the   |   |   |
|                    |   |   | maximum possible points for the Item. The  |   |   |
|                    |   |   | Examiners are identified by number from    |   |   |
|                    |   |   | left to right across the top of the chart. |   |   |
|                    |   |   | Their individual scores for the Items are  |   |   |
|                    |   |   | given in the column below their number,    |   |   |
|                    |   |   | with their total score at the bottom. The  |   |   |
|                    |   |   | range (low--high) of scores for an Item is |   |   |
|                    |   |   | identified by scanning across a line. The  |   |   |
|                    |   |   | next five columns show summary data for    |   |   |
|                    |   |   | each Item: the average percent score; the  |   |   |
|                    |   |   | median point score; the median percent     |   |   |
|                    |   |   | score; and the standard deviation. Along   |   |   |
|                    |   |   | the bottom row for each Category, the      |   |   |
|                    |   |   | table provides the maximum possible        |   |   |
|                    |   |   | Category points and the total Category     |   |   |
|                    |   |   | points assigned by each Examiner.          |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | Before assigning an Examiner               |   |   |
|                    |   |   | responsibility for an Item, scan the       |   |   |
|                    |   |   | scoring data. Typically, Examiners are     |   |   |
|                    |   |   | assigned leads on Items for which their    |   |   |
|                    |   |   | scores were at or near the median of Stage |   |   |
|                    |   |   | 1 scoring.                                 |   |   |
+--------------------+---+---+--------------------------------------------+---+---+
| Stage 1 Scoring    |   |   | The **Stage 1 Scoring Graph** is a box     |   |   |
| Graph              |   |   | plot graphic representation of the tabular |   |   |
|                    |   |   | data presented in the Stage 1 Examiner     |   |   |
|                    |   |   | Scoring Table. The box plot displays the   |   |   |
|                    |   |   | distribution of the scores for all Items   |   |   |
|                    |   |   | assigned by Examiners in Stage 1, as well  |   |   |
|                    |   |   | as summaries of median and quartile        |   |   |
|                    |   |   | statistics describing the distribution.    |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | The large box indicates the center of the  |   |   |
|                    |   |   | distribution and it contains the middle    |   |   |
|                    |   |   | 50% of the scores. The bottom of the box   |   |   |
|                    |   |   | is drawn at the first quartile and the top |   |   |
|                    |   |   | at the third quartile.                     |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | Scores that are far from the middle of the |   |   |
|                    |   |   | distribution and that meet certain         |   |   |
|                    |   |   | statistical definitions (defined below)    |   |   |
|                    |   |   | are called "outliers" and "extremes."      |   |   |
|                    |   |   | "Outliers" are represented by a small      |   |   |
|                    |   |   | circle (o), and "extremes" are represented |   |   |
|                    |   |   | by an asterisk (\*).                       |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | The median score (middle score or 50^th^   |   |   |
|                    |   |   | percentile) appears as a small square (). |   |   |
|                    |   |   | The median is not necessarily halfway      |   |   |
|                    |   |   | between the first and third quartile. A    |   |   |
|                    |   |   | line connects the median from Item to      |   |   |
|                    |   |   | Item.                                      |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ###                                        |   |   |
|                    |   |   | #### The lines extending above and below t |   |   |
|                    |   |   | he box are known as whiskers. They indicat |   |   |
|                    |   |   | e the upper and lower range (minimum and m |   |   |
|                    |   |   | aximum) of scores that lie outside the box |   |   |
|                    |   |   | , excluding the "outliers" and "extremes." |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ####### This graph identifies              |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | 1.  *the range of scores on each Item*     |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ```{=html}                                 |   |   |
|                    |   |   | <!-- -->                                   |   |   |
|                    |   |   | ```                                        |   |   |
|                    |   |   | 1.  *the median score on each Item*        |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ```{=html}                                 |   |   |
|                    |   |   | <!-- -->                                   |   |   |
|                    |   |   | ```                                        |   |   |
|                    |   |   | 2.  *where the middle 50% of the scores    |   |   |
|                    |   |   |     fall for an Item and how closely       |   |   |
|                    |   |   |     grouped scores are*                    |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | 3.  *scores that are statistical           |   |   |
|                    |   |   |     "outliers" and "extremes"*             |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | ####### Statistical definitions:           |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | 1.  *The **interquartile range** (IQR), is |   |   |
|                    |   |   |     the range between the first and third  |   |   |
|                    |   |   |     quartiles, Q~3\ ---~Q~1~.*             |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | 2.  ***Outliers** are points that fall     |   |   |
|                    |   |   |     more than a distance of  1.5 times    |   |   |
|                    |   |   |     the interquartile range from the box   |   |   |
|                    |   |   |     ends of Q~3~ and Q~1~ (Q~3~ + 1.5 IQR  |   |   |
|                    |   |   |     and Q~1~ -- 1.5 IQR) and less than 3   |   |   |
|                    |   |   |     times that distance (Q~3~ + 3 IQR and  |   |   |
|                    |   |   |     Q~1~ - 3 IQR). **Extremes** are points |   |   |
|                    |   |   |     that fall more than 3 times the        |   |   |
|                    |   |   |     distance from the box (\> Q~3~ + 3 IQR |   |   |
|                    |   |   |     and \< Q~1~ - 3 IQR). These distances  |   |   |
|                    |   |   |     are based on median values and should  |   |   |
|                    |   |   |     not be confused with standard          |   |   |
|                    |   |   |     deviations, which are based on mean    |   |   |
|                    |   |   |     values*.                               |   |   |
+--------------------+---+---+--------------------------------------------+---+---+
| Stage 1 Frequency  |   |   | The **Stage 1 Frequency of Scores** is a   |   |   |
| of Scores          |   |   | table showing how many Stage 1 Examiners   |   |   |
|                    |   |   | selected a specific Item percent score.    |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | The horizontal axis at the bottom shows    |   |   |
|                    |   |   | the Item numbers. The vertical axis shows  |   |   |
|                    |   |   | the percentage scores. Within the grid     |   |   |
|                    |   |   | boxes is a frequency count of the number   |   |   |
|                    |   |   | of Examiners who scored an Item at each of |   |   |
|                    |   |   | the 5% increments.                         |   |   |
|                    |   |   |                                            |   |   |
|                    |   |   | *Note*: For ease in reading, zero          |   |   |
|                    |   |   | frequency counts are not recorded.         |   |   |
+--------------------+---+---+--------------------------------------------+---+---+

![](media/image3.wmf)

![](media/image4.wmf)

![](media/image5.wmf)

+--------------------+---+----------------------------------------------+
| **Preparing the    |   |  The team leader faxes the consensus Score  |
| Consensus          |   | Summary Worksheet to ASQ immediately upon    |
| Scorebook**        |   | completion of the consensus call.            |
|                    |   |                                              |
|                    |   |  The team leader works with team members to |
|                    |   | ensure that the revised Item worksheets and  |
|                    |   | other worksheets reflect the consensus call  |
|                    |   | discussions and meet the Comment Guidelines  |
|                    |   | and Criteria requirements.                   |
|                    |   |                                              |
|                    |   |  The scorebook editor revises and the team  |
|                    |   | leader approves the revisions to the         |
|                    |   | consensus scorebook **comments** (strengths  |
|                    |   | and OFIs) and **Key Themes Worksheet**. The  |
|                    |   | scorebook editor will                        |
|                    |   |                                              |
|                    |   | -   *clarify and standardize the language    |
|                    |   |     among Items*                             |
|                    |   |                                              |
|                    |   | -   *eliminate any conflicts between         |
|                    |   |     strengths and OFIs within and among      |
|                    |   |     Items*                                   |
|                    |   |                                              |
|                    |   | -   *check that all comments conform to the  |
|                    |   |     Comment Guidelines (contain a single,    |
|                    |   |     complete thought; avoid jargon and are   |
|                    |   |     not prescriptive, predictive or          |
|                    |   |     negative; include specific examples and  |
|                    |   |     figure references and focus on major     |
|                    |   |     objectives of the Criteria, etc.)*       |
|                    |   |                                              |
|                    |   | -   *ensure that the Key Themes Worksheet    |
|                    |   |     conforms to the required format          |
|                    |   |     (contains the most significant strengths |
|                    |   |     and concerns and puts them into          |
|                    |   |     perspective, taking into account Item    |
|                    |   |     linkages, key factors, and core values)* |
|                    |   |                                              |
|                    |   | > (Site visit issues and the Key Factors     |
|                    |   | > Worksheet will be revised if the applicant |
|                    |   | > is selected to go on to site visit.)       |
|                    |   |                                              |
|                    |   |  The scorebook editor assembles the         |
|                    |   | consensus scorebook **(paper copy and        |
|                    |   | diskette for word processed pieces),**       |
|                    |   | consisting of the following:                 |
|                    |   |                                              |
|                    |   | -   *Key Factors Worksheet*                  |
|                    |   |                                              |
|                    |   | ####### *Key Themes Worksheet*               |
|                    |   |                                              |
|                    |   | -   *Item Worksheets containing key factors  |
|                    |   |     and all **comments** (strengths, OFIs,   |
|                    |   |     Areas to Address \[e.g., a(1), b(3)\],   |
|                    |   |     ADLI) **and site visit issues, if        |
|                    |   |     applicable.***                           |
|                    |   |                                              |
|                    |   | -   *Score Summary Worksheet (previously     |
|                    |   |     faxed to ASQ by the team leader)*        |
|                    |   |                                              |
|                    |   |  The scorebook editor sends the revised     |
|                    |   | consensus scorebook (paper copy and diskette |
|                    |   | for word processed pieces) to the team       |
|                    |   | leader.                                      |
+--------------------+---+----------------------------------------------+
| Preparing the      |   |  The team leader reviews, revises, and      |
| Consensus          |   | approves the consensus scorebook (legible    |
| Scorebook          |   | pen and ink changes are acceptable).         |
|                    |   |                                              |
| ***-continued***   |   |  The team leader sends the **final, edited, |
|                    |   | feedback-ready consensus scorebook** (paper  |
|                    |   | copy and diskette for word processed pieces) |
|                    |   | **for receipt by NIST/BNQP no later than     |
|                    |   | September 14.**                              |
+--------------------+---+----------------------------------------------+

***Section Seven***

***Materials From the Examiner Handbook and More FAQs***

**7.0 STAGE 2, CONSENSUS REVIEW**

**Overview**

Applicants selected by the Panel of Judges for further evaluation move
to Stage 2, Consensus Review. The purpose of Stage 2 is to clarify and
resolve differences in

individual Examiner observations and scoring from Stage 1, Independent
Review. During this stage, a team of Examiners reaches consensus on key
themes, KFs, comments that capture the team's collective view of the
applicant's strengths and OFIs, and the resulting score. The consensus
scores and scoring profiles are used by the Panel of Judges in selecting
applicants to be site visited. If an applicant is not selected for a
site visit, the team's comments are used for the final scorebook. The
final scorebook is the basis for the feedback report to the applicant.

**Composition of the Consensus Team**

A Consensus Team consists of a combination of Examiners and Senior
Examiners. Whenever possible, team members are selected from Examiners
who completed the Stage 1, Independent Review, of the application. Team
leaders are selected from Senior Examiners who have received additional
training on the Award's purposes and processes.

**Consensus Review Process**

Consensus is an agreed-upon decision on a scorebook comment and
numerical score, based upon the contributions of ***all*** team members.
NIST will assign two Senior Examiners to serve as team leader and backup
team leader. The team leader will create a schedule for and coordinate
all tasks associated with the team's successful completion of the
consensus review process for its applicant. The Consensus Team, via
conference calls, reaches consensus on comments synthesized from the
Stage 1 scorebooks and arrives at a consensus score for each Criteria
Item. As a member of the team, the scorebook editor prepares a consensus
scorebook using the work of the Consensus Team. The consensus scorebook
is used as the basis for the feedback report for applicants dropping out
at Stage 2. It is also the basis for site visit planning for applicants
going on to Stage 3.

The key steps in the consensus review process for Examiners are as
follows:

-   Receive assignment to a Consensus Team.

-   Complete planning/prework.

```{=html}
<!-- -->
```
-   Review all Stage 1 scorebooks.

-   Draft the Key Factors Worksheet and the Key Themes Worksheet.

-   Develop draft Item Worksheets for the assigned Items (including
    proposed comments and score), complete any other assignments, and
    distribute to other team members and NIST.

-   Serve as Item backup as assigned.

-   Review all draft Item Worksheets and assignments prepared by other
    team members before the consensus call(s).

```{=html}
<!-- -->
```
-   Plan the discussion order of the consensus conference call(s).

-   Conduct the conference call(s).

```{=html}
<!-- -->
```
-   Lead discussion on assigned Items.

-   Finalize the Key Factors Worksheet and the Key Themes Worksheet.

-   Perform other roles as assigned.

```{=html}
<!-- -->
```
-   Complete consensus call follow-up tasks.

```{=html}
<!-- -->
```
-   Revise the assigned Item Worksheets to reflect discussion on the
    consensus call(s) and forward to the team leader and scorebook
    editor.

```{=html}
<!-- -->
```
-   Prepare peer reviews of Stage 1 scorebooks and forward to NIST.

-   Prepare and submit the final consensus scorebook, if assigned.

-   Review consensus scores/scoring profiles and selection of applicants
    for site visit review by the Panel of Judges.

**Note:** Examiners selected for Consensus Teams will receive detailed
instructions on each step of the consensus review prior to beginning the
process.

**Conducting the Consensus Call**

***Determining Consensus Comments and Consensus Scores***

-   The team discusses Criteria requirements and KFs for each specific
    Item.

-   The team discusses the applicant Item by Item, noting KFs and key
    themes. The team discusses the comments on the draft Item Worksheets
    prepared by the Category/ Item leader. The discussion focuses on how
    to tailor the individual comments so they will provide usable
    information to the applicant. Divergent views are discussed and
    resolved by going back to the application, KFs, Criteria
    requirements, and Scoring Guidelines. The team identifies important
    Criteria requirements that are not already addressed in the draft
    comments and develops comments representing its view of the
    applicant on these issues.

-   The team reaches consensus on the Item comments, ensuring that all
    comments represent the team's commonly held view of the applicant's
    strengths and OFIs as each relates to the main objectives of the
    Criteria for the Item.

```{=html}
<!-- -->
```
-   Using the Scoring Guidelines, the assigned Category/Item discussion
    leader will propose a scoring range and then a consensus Item score
    aligned with the agreed upon comments. Team members are polled to
    determine agreement with the proposed range and the score.

-   If consensus cannot be reached and the difference in proposed
    consensus scores among team members participating on the call is 30
    percent or less, the average (arithmetic mean) of the proposed
    consensus scores of team members participating in the conference
    call may be used.

-   The team discusses and agrees upon the key themes resulting from the
    application review.

**Frequently Asked Questions About Stage 2, Consensus Review**

**1. How many Examiners will participate in Stage 2?**

Generally, one-half to two-thirds of the Examiners who complete a Stage
1 review will receive a Consensus Team assignment for that applicant,
providing the applicant moves forward in the review process. But should
there be an opening on a Consensus Team, an Examiner from Stage 1 whose
applicant did not move forward could be asked to fill the opening. The
Examiner is then required to complete the Key Themes Worksheet for this
applicant in order to participate in the consensus review.

**2. Can the draft Item Worksheets be faxed to other team members?**

Yes. Consensus work documents may be faxed; however, all communications
should meet confidentiality requirements. (See pages 3-3 through 3-4,
Rules of Conduct and Code of Ethical Standards.) The applicant should be
referred to by number only. Recipients of the faxed document should be
called first so they can protect confidentiality by receiving the
document personally.

**3. When are conference calls typically scheduled?**

Examiners are notified of the consensus calendar during the Examiner
Preparation Course and are encouraged to plan their schedules to allow
for participation on a Consensus Team. Scheduling the conference calls
can be one of the most difficult parts of the consensus review.
Therefore, Examiners are asked to clear their calendars during the
assigned times in order to be available to participate in the scheduled
calls. It is important to be flexible. Most calls take place during
normal business hours. Some teams find evenings and weekends to be the
only time they can schedule the calls. It is not unusual for team
members to participate on consensus calls during business travel or
vacation leave.

**4. What if an Examiner cannot be there for part of the conference
call?**

It is critical that ***all*** team members participate in the consensus
review process. It is essential that each team member participate in the
***entire*** conference call. Team members should contact the team
leader immediately if a problem arises with the proposed schedule. If
schedules change and an Examiner must step away from the call, the team
leader will have to reschedule the call to allow for the full
participation of that team member.

**5. Can electronic mail be used to forward draft Item Worksheets to
other team members?**

No. Electronic mail via the Internet is not a secure means of
communication for the purpose of applicant review and should not be
used. Instead, use overnight mail or a secure fax. The use of cellular
and cordless telephones for consensus calls is also not a secure means
of communication and, thus, is not permitted. Although there are methods
to secure these communications, there are no guarantees.

**6. Can secretaries, family members, or anyone else help prepare and
transmit consensus review documents?**

No. Secretaries and other persons should not prepare, copy, or transmit
confidential consensus review documents. Examiners are solely
responsible for all the materials and information.

**7. What are the time commitments required for participating in the
consensus review process?**

The consensus review process occurs from mid-August through
mid-September. It is estimated that 20--27 hours will be required: 2--3
hours for the planning call, 12--16 hours in preparation prior to the
consensus calls, 10--12 hours for the consensus calls, and 2--3 hours
after the calls to rewrite the draft Item Worksheets and other consensus
scorebook documents.

***Consensus Review: DOs and DON'Ts***

-   *Do* be flexible during call scheduling.

```{=html}
<!-- -->
```
-   *Do* prepare. It is essential that each team member fully prepare
    **prior** to the conference call. Complete the draft Item Worksheets
    by considering the comments from all scorebooks and synthesizing the
    information into the most significant four to six KFs for the
    related Item, and six to ten comments most important to the
    applicant and respond to the main points of the Criteria.
    **Preparation for the call generally takes more time than the call
    itself.**

```{=html}
<!-- -->
```
-   *Do* distribute the draft Item Worksheets to other team members and
    NIST in advance to enhance the efficiency and effectiveness of the
    consensus conference call.

-   *Do* review all draft Item Worksheets prepared by other team members
    prior to the call. Be prepared to discuss.

-   *Do* review the draft Key Factors Worksheet and Key Themes Worksheet
    prepared by the scorebook editor prior to the call. Be prepared to
    discuss.

-   *Do* meet deadlines and connect to the conference calls on time.

-   *Do* have the necessary materials at hand: the appropriate Criteria,
    the application, the Scoring Guidelines, all Stage 1 Scorebooks, and
    the draft Key Factors, Key Themes, and Item Worksheets from other
    team members.

-   *Do* participate in the conference call.

```{=html}
<!-- -->
```
-   Listen to all points of view presented, including those of Examiners
    who prepared independent reviews but are not members of the
    Consensus Team.

-   Provide your point of view.

-   Focus on discussing the substance of the comments, the Criteria, and
    the Scoring Guidelines, and then align the score with the comments.

```{=html}
<!-- -->
```
-   *Do* revise the Key Factors Worksheet and Key Themes Worksheet based
    on the conference call discussion.

-   *Do* provide all written work in accordance with the team leader's
    instructions.

-   *Don't* focus solely on numerical scores.

-   *Don't* isolate or polarize team members who have differing views.

-   *Don't* become defensive about your scores from the independent
    review. Remember, one of the outcomes of Stage 2 is a score that
    reflects the entire team's collective evaluation.

-   *Don't* ask other team members to justify high/low scores from the
    independent review.

ADDITIONAL FREQUENTLY ASKED QUESTIONS

ABOUT THE

**CONSENSUS REVIEW STAGE**

**These questions and answers supplement those in the Examiner Handbook,
reprinted in Section Seven of the Consensus Manual.**

(1) **How are team members selected for Consensus Teams?**

The Consensus Team is composed primarily of Examiners who read the
application at Stage 1. In addition, we consider proper team balance in
terms of Criteria knowledge, Award process experience, industry/sector
expertise, and Category expertise. To be selected, Examiners must also
be available to participate in the entire Consensus review process and,
preferably, in the Stage 3 site visit review process if the applicant
moves on.

(2) **This is my first time on a Consensus Team. What are the different
    roles and assignments for team members?**

There are several standard roles that Consensus Team members perform
throughout the entire review process. These roles are team leader,
backup team leader, Category/Item lead and backup, computer expert, and
scorebook editor. There are also other roles that team members play
while on the consensus calls. These roles include scorebook sponsor,
timekeeper, scorekeeper, criteria expert, and process checker. Each of
these roles is covered in detail in Section Three of the Consensus
Manual.

(3) **I just found out that my Stage 1 applicant is going on to
    consensus. What are my first tasks?**

Congratulations! Your first task is to provide your team leader with
your biography and the dates that you are available (and unavailable)
for the planning call and consensus calls. Your team leader will be
contacting you in early August to request this information.

(4) **I am a newly added member of a Consensus Team. I did not review
    the application during the Stage 1 Independent Review. What do I
    need to do to prepare for the consensus review process?**

If you are a newly added member of a Consensus Team, you will receive a
package from ASQ containing the application and other materials. (This
package is similar to the one that you received during the Stage 1:
Independent Review.) Once you receive this package, you should read the
application and prepare a complete set of key themes for the applicant
(i.e., complete the Key Themes Worksheet). However, you [do
not]{.underline} need to prepare a complete scorebook, nor do you need
to score the application, as you did in the Stage 1 review. The package
from ASQ and your team leader will provide additional information on
your specific tasks.

(5) How are site visit issues selected?

Your team leader will develop a plan for site visit issues. Your team
leader will inform you if you are to prepare site visit issues.
Typically, 2-4 site visit issues are prepared for each of your assigned
Categories/Items. The issues are based on the need to verify strengths
or clarify OFIs contained in the consensus scorebook. Since the site
visit is not an audit, the selection of issues or questions for
clarification and/or verification during the site visit is crucial.

Characteristics of a well-chosen site visit issue include the following:

-   *[It is an essential component of the score]{.underline}.* It
    focuses on factors that are important for the applicant, even if it
    is not important for all organizations. It may be one of the
    significant strengths or OFIs that requires verification (e.g., the
    extent of deploy­ment), or it may be clarification of a significant
    issue for which information in the application is missing or
    unclear.

-   *[It is "cross-cutting."]{.underline}* It is the type of issue that
    affects, directly or indirectly, more than one Criteria Item. Also,
    it may include the degree to which an aspect of manage­ment is
    integrated throughout the applicant's organization.

-   *[It is part of the deployment determination.]{.underline}* The
    degree to which the applicant's approaches are deployed is often
    difficult to assess from the written application. The degree of
    deployment is often dependent upon the maturity of the applicant.

-   *[It is verifiable]{.underline}.* Acceptable techniques by which the
    team can get an answer to the issue are exam­in­­ing data sources,
    interviewing employees, or listening to presentations by the
    applicant. Unac­ceptable techniques include (1) interviewing
    customers, students, or patients unless they are a part of an
    official manage­ment or advisory structure; (2) conducting impromptu
    surveys; and (3) assessing individual customer complaints.

(6) **I have been on some very long consensus calls. What strategies
    might reduce the amount of time that the calls take?**

Consensus calls are, by design, typically 8--10 hours (two calls of 4--5
hours each). To avoid longer calls, it is imperative that your team meet
its deadlines during the planning phase prior to the calls. This means
that Item Worksheets must be exchanged and reviewed in a timely manner
and that they must be distributed to all team members well in advance of
the call. It also means that all team members must review the worksheets
prior to the call and be ready to discuss them.

Additionally, while on the call, Category/Item leads must present their
Items efficiently and effectively. This means that Item leads should
rehearse delivering their Items prior to the call and be prepared to
summarize the Item requirements, key factors, and comments. (They should
not read their Item Worksheets verbatum.) They should also be prepared
to discuss any divergent views and how these differences were resolved.
A model script for presenting an Item is available in Section Six of the
Consensus Manual.

(7) **I participated in the consensus review stage last year. What's new
    and different this year?**

First, some similarities: Like last year, this year's Consensus Teams
have only six team members, as opposed to seven in some prior years.
With only six members, the role of scorebook editor will be performed by
a team member who also is a Category/Item lead. We are again focusing on
keeping calls to 8--10 hours or less. The calls are designed to be
completed in this timeframe, but in recent years, many calls have
exceeded these limits. We will be working with each of the teams to
reduce the total call time.

As for changes:

(1) drafting and discussing site visit issues and reaching consensus on
    them will be optional at Stage 2. Each team leader will decide if
    the team will prepare site visit issues for each item during the
    consensus process.

(2) For the first time, there will be a consensus hotline. By calling
    the hotline, a team can obtain Criteria and procedural assistance
    during consensus calls.

(3) The NIST/BNQP monitor may not listen to the entire consensus call,
    because the teams can obtain assistance by calling the hotline.

```{=html}
<!-- -->
```
(8) **I still have questions. Where can I get additional information
    about the consensus review process?**

You may still have questions after reviewing the *Consensus Manual* and
other materials provided by your team leader and ASQ. If this is the
case, you should contact your team leader or the NIST monitor who is
assigned to your team. Their names and contact information can be found
in the team roster that you were provided as part of your consensus
package.

**Index of Key Terms**

agenda, 15, 16, 19, 20, 24, 40, 43, 56, 57, 59

agenda for the planning call, 3

American Society for Quality (ASQ), 3, 4, 15, 16, 17, 18, 19, 20, 23,
25, 34, 36, 38, 39, 40, 41, 42, 43, 45, 56, 57, 58, 59, 62, 66, 85, 94,
96

assignments, 3, 4, 15, 16, 17, 18, 19, 20, 21, 36, 40, 43, 56, 59, 90,
91, 94

backup file, 24

backup team leader, 8, 16, 14, 18, 21, 27, 40, 90, 94, see also roles

biography, 38, 41, 42, 45, 53, 54

biography form, 36, 43

call-in time, 57

category/item backup, 15, 16, 14, 15, 24, 60, 64, see also roles

category/item lead, 15, 16, 14, 15, 60, 77, 94, 96, see also roles

category cop, 15, see also roles

cellular and portable phones, 36, 40, 43

checklist for process checker, 76

chronology of activities, 17, 18, 19, 20, 21, 22, 23

comment guidelines, 14, 15, 16, 23, 30, 32, 38, 39, 41, 42, 45, 60, 69,
85

feedback-ready, 4, 23, 58, 64, 86

scorebook comments, 85

communication among team members, 48

computers

compatibility, 26

computer expert, 15, 16, 15, 56, 20, 94, see also roles

computer files, 24, 26

computer use, 15, 20, 25, 26, 57

computer viruses, 26

electronic files, 24, 25

file labeling conventions, 15

file naming conventions, 26

format requirements, 26

NIST requested format, 17

instructions for planning computer use, 15

saving protocol, 57

shared files, 26

software recovery programs, 25

conference call forms, 16

confidentiality, 20, 24, 36, 40, 43, 59, 60, 61, 92

information transfer regarding the applicant, 58

consensus, 1, 3, 6, 8, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 27, 29,
36, 38, 39, 40, 41, 43, 45, 52, 54, 56, 58, 60, 61, 62, 64, 65, 66, 67,
69, 75, 77, 78, 85, 86, 90, 91, 92, 93, 94, 95, 96

consensus call, 4, 15, 16, 15, 19, 20, 21, 23, 24, 29, 58, 59, 60, 61,
62, 64, 77, 85, 91, 96

conducting the consensus call, 14, 91

consensus hotline, 16, 62, 65, 96

immediately following the consensus calls, 21

planning calls, 3, 36, 40, 43

prior to the consensus call, 20

presentation style (category/item discussion), 15

roll call, 20, 22

consensus assignments, 78

consensus instructions, 3

consensus hotline, 16, 62, 65, 96

consensus review process, 8, 90

consensus review timeline, 1, 3, 19, 23

consensus score, 61, 90

consensus scorebook, 14, 23, 24, 38, 85, 86, see also scorebook

consensus scoring, 15, 22, 77

consensus strategy, 19

consensus team, 3, 8, 14, 15, 16, 18, 24, 27, 29, 36, 40, 43, 54, 60,
65, 90, 92, 93, 94

consensus timeline, 18

criteria cop, 8, 15, 16, 15, 20, 57, see also roles

dates unavailable form, 51, 52

dates unavailable form: consensus calls, 43

dates unavailable form: planning call, 42

draft correspondence, 19

draft item worksheets, 16, 24

electronic files, 24, 25

e-mail, 20, 24, 36, 38, 40 41, 43 45, 57, 58, 59

ethics, 15, 16, 74

examiner scoring table, 17, 38, 41, 45, 78, 79

fax machine, 24, 36, 40, 43

secure fax, 24, 58, 60, 61, 92

feedback-ready, 4, 23, 58, 64, 86

file labeling convention, 15

file naming convention, 26

final item worksheets, 16

format requirements, 26

forms, 15, 16, 17, 18, 20, 29, 30, 31, 32, 38, 41, 42, 45, 53, 54, 59,
62, 64, 66

updated dates unavailable forms, 41

frequency of scores, 17, 78, 80

frequently asked questions (FAQS), 88, 92

ground rules, 15, 19, 20, 22, 36, 40, 43, 57

handbook, 88, 94

hotline, 4, 15, 23, 96

independent review, 8, 38, 41, 45, 90, 94

information transfer regarding the applicant, 58

instructions for planning computer use, 15

intact team, 38, 64

non-intact team, see original read, new read, new team member

key factors worksheet, 16, 14, 20, 22, 30, 32, 57, 61, 85, 90, 91, 93

key themes, 3, 8, 16, 14, 17, 20, 22, 23, 30, 32, 36, 38, 40, 41, 43,
45, 56, 57, 59, 65, 85, 90, 91, 92, 93, 94

key themes worksheet, 17, 20, 38, 43

letters, 29, 32, 34, 36, 40, 43

letter to backup team leaders, 34

letter to team leaders, 32

letter to team members, 36

new read, 36, 38, 40, 41, 43, 45

new team members, 36, 38, 40, 41

NIST requested format, 17

NIST monitor, 16, 19, 20, 21, 22, 59, 65, 96

options for developing site visit issues, 14, 19, 27

original read, 36, 40, 43

page format/layout, 26

PC compatible, 57

peer evaluation

peer evaluation form, 29

peer evaluation review process, 28, 30

percent score, 16, 77, 78, 80

phantom scorebooks, 16, 18

planning and prework, 14

planning calls, 3, 36, 40, 43

preparing the consensus scorebook, 74

presentation style (category/item discussion), 15

prework, 14, 66, 90

process checker, 8, 15, 16, 15, 20, 22, 56, 76,.94, see also roles

process, description of consensus, 6

proposed scoring range, 77

purpose, of consensus, 8

reconnection policy, 20, 59

review schedule, 64

roles, 8, 14, 22, 65, 91, 94

backup team leader, 8, 16, 14, 18, 21, 27, 40, 90, 94

category cop, 15

category/item backup, 15, 16, 14, 15, 24, 60, 64

category/item lead, 15, 16, 14, 15, 60, 77, 94, 96

computer expert, 15, 16, 15, 56, 20, 94

criteria cop, 8, 15, 16, 15, 20, 57

process checker, 8, 15, 16, 15, 20, 22, 56, 76, 94

checklist for process checker, 76

scorebook editor, 3, 4, 8, 15, 16, 14, 18, 19, 20, 23, 24, 25, 57, 58,
59, 62, 64, 85, 90, 91, 93, 94, 95

scorebook sponsor, 8, 15, 16, 18, 56, 94

scorekeeper, 8, 15, 16, 20, 22, 77, 94

team leader, 14

timekeeper, 8, 15, 16, 20, 22, 57, 94

roles and responsibilities, 14, 17

roll call, 20, 22

rules, 15, 16, 57, 65, 77, 92

sample communications, 19

sample completed biography form, 45

sample consensus call agendas, 55

sample consensus call process, 51

sample consensus review schedule, 54

sample item worksheet, 64

sample peer evaluation letter, 27

sample planning call agenda, 50

sample stage 1 examiner scoring table, 71

sample stage 1 frequency of scores, 73

sample stage 1 scoring graph, 72

sample team leader planning letter, 47

sample team leader thank you note, 56

sample script---presentation of item 1.1, 61

saving protocol, 57

scoring

percent score, 16, 77, 78, 80

proposed scoring range, 77

score, 8, 16, 15, 22, 30, 32, 60, 61, 77, 78, 79, 90, 91, 93, 94, 95

scoring graph, 17, 38, 41, 45, 78, 79

scoring range, 16, 14, 15, 22, 60, 61, 69, 73, 77, 91

scoring guidelines, 22, 38, 39, 41, 42, 45, 57, 60, 91, 93

stage 1 examiner scoring table, 17, 38, 41, 45, 78

stage 1 frequency of scores, 78, 80

stage 1 frequency table, 38, 41, 45

stage 1 scoring graph, 17, 38, 41, 45, 78, 79

score summary worksheet, 4, 15, 17, 22, 23, 56, 62, 85

scorebook, 3, 4, 8, 15, 16, 14, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 29, 30, 31, 36, 38, 41, 42, 57, 58, 59, 60, 61, 62, 64, 69, 85, 86,
90, 91, 93, 94, 95

preparing the consensus scorebook, 74

scorebook comments, 85

scorebook editor, 3, 4, 8, 15, 16, 14, 18, 19, 20, 23, 24, 25, 57, 58,
59, 62, 64, 85, 90, 91, 93, 94, 95, see also roles

scorebook editor responsibilities, 23, 24

scorebook sponsor, 8, 15, 16, 18, 56, 94, see also roles

scorekeeper, 8, 15, 16, 20, 22, 77, 94, see also roles

scoring graph, 17, 38, 41, 45, 78, 79

scoring guidelines, 22, 38, 39, 41, 42, 45, 57, 60, 91, 93

scoring range, 16, 14, 15, 22, 60, 61, 69, 73, 77, 91

secure fax, 24, 58, 60, 61, 92

shared files, 26

site visit issues, 27, 72

options for developing site visit issues, 14, 19, 27

site visit planning, 8, 23, 27, 90

site visit schedule, 22

software recovery programs, 25

stage 1 examiner scoring table, 17, 38, 41, 45, 78

stage 1 frequency of scores, 78, 80

stage 1 frequency table, 38, 41, 45

stage 1 scoring graph, 17, 38, 41, 45, 78, 79

steps for creating draft consensus item worksheet, 59

synthesizing a stage 2 comment, 69, 70, 93

team biographies, 19

team leader, 14, see also roles

team leader responsibilities, 17

team leader thank you note, 66

team leader to team members (sample correspondence), 55

team lists, 3, 17

team members to team leader (sample correspondence) 50

team roster, 96

telephone, 17, 20, 58

cellular and portable phones, 36, 40, 43

reconnection policy, 20, 59

timekeeper, 8, 15, 16, 20, 22, 57, 94, see also roles

tools, 67, 78

updated dates unavailable forms, 41

virus detection software, 26

word processing formats, 24

worksheets, 3, 16, 14, 16, 17, 19, 20, 21, 23, 24, 26, 30, 32, 36, 38,
40, 41, 45, 57, 60, 61, 62, 64, 85, 90, 91, 92, 93, 95

final item worksheet, 16

key factors worksheet, 16, 14, 20, 22, 30, 32, 57, 61, 85, 90, 91, 93

key themes worksheet, 17, 20, 38, 43

sample item worksheet, 64

score summary worksheet, 4, 15, 17, 22, 23, 56, 62, 85

steps for creating a draft consensus item worksheet, 59

**Prepared by:**

**Award Process Team**

**Baldrige National Quality Program**

**National Institute of Standards and Technology**