Step-by-Step Instructions for CONSENSUS REVIEW

  -------------------- --------------------------------------------------
  **Goal and Purpose** The ultimate goal of the Consensus Review process
                       is to provide each Award applicant with a report
                       detailing areas of strength and opportunities for
                       performance improvement. The purpose of Consensus
                       Review is to create the basis for this
                       report---one scorebook based on the requirements
                       of the Criteria for Performance Excellence. This
                       Consensus Scorebook, comprising comments and
                       scores, represents the collective knowledge and
                       analytical skills of a team of Examiners using a
                       systematic review process**.**

  **Roles**            Each team member (1) serves as lead for one or
                       more Items, synthesizing feedback from the entire
                       team and developing comments; (2) serves as a
                       backup for one or more Items, providing feedback
                       to an Item lead; and (3) participates in
                       consensus-building discussions---both virtually
                       and through a consensus call---to discuss and
                       agree on the comments and scoring that the
                       applicant ultimately receives in the form of a
                       feedback report.
  -------------------- --------------------------------------------------

+-------------------+--------------------------------------------------+
| **STEPS**         | **ACTIONS**                                      |
+===================+==================================================+
| 1.  Complete the  | -   You must complete an Independent Review      |
|     Independent   |     Scorebook to participate in the Consensus    |
|     Review        |     Review. (The only exceptions are Alumni      |
|     Scorebook.    |     Examiners, who may be added to the team late |
|                   |     in the review process.)                      |
+-------------------+--------------------------------------------------+
| 2.  Participate   | -   The Team Leader sets the date and time for   |
|     in planning   |     the first planning call. Participation in    |
|     call 1.       |     this call is essential for understanding the |
|                   |     entire timeline, the details of the process, |
|                   |     and the ground rules for working with the    |
|                   |     team.                                        |
+-------------------+--------------------------------------------------+
| 3.  Download the  | -   A few hours before call 2, the Team Leader   |
|     key factors   |     posts a Key Factors Worksheet to             |
|     from          |     *examinerdepot*. Before the call, carefully  |
|                   |     review this worksheet and note recommended   |
|  *examinerdepot.* |     changes or additions.                        |
+-------------------+--------------------------------------------------+
| 4.  Participate   | -   The Team Leader finalizes consensus          |
|     in planning   |     assignments, asks for feedback on the key    |
|     call 2.       |     factors, describes the process and next      |
|                   |     steps in the consensus process, and answers  |
|                   |     questions from team members.                 |
+-------------------+--------------------------------------------------+
| 5.  Download and  | -   From the BNQP Examiner Learning Resources    |
|     save the      |     Center                                       |
|     correct blank |     (<http:/                                     |
|     Consensus     | /www.baldrige.nist.gov/Examiner_Resources.htm>), |
|     Review Item   |     click on the *Consensus Review Scorebook,*   |
|     Worksheet for |     and download the Consensus Review Item       |
|     each assigned |     Worksheets you need. Each worksheet posted   |
|     Item.         |     there (named 1.1, 1.2, and so on) contains a |
|                   |     rationale sheet, Item worksheet, and scoring |
|                   |     grid.                                        |
|                   |                                                  |
|                   | > Also posted are Conflict of Interest and Code  |
|                   | > of Conduct forms (for new team members to      |
|                   | > complete and post), the Key Factor/Key Theme   |
|                   | > Worksheets, the Score Summary Worksheet, and   |
|                   | > the entire scorebook structure, which includes |
|                   | > placeholders for each worksheet.               |
|                   |                                                  |
|                   | -   Save your Consensus Review Item Worksheets   |
|                   |     using the file-naming convention specified   |
|                   |     on page 1 of the worksheet file.             |
+-------------------+--------------------------------------------------+
| 6.  From          | -   Download these Independent Review Worksheets |
|                   |     from your applicant site on examinerdepot,   |
|  *examinerdepot*, |     within the Independent Review link. Make     |
|     isolate all   |     sure to include the ideas from "phantom"     |
|     Independent   |     scorebooks, if any.                          |
|     Review        |                                                  |
|     Worksheets    | > *Phantom scorebooks are Independent Review     |
|     for your      | > Scorebooks developed by Examiners who do not   |
|     assigned      | > participate in Consensus Review.*              |
|     Item.         |                                                  |
|                   | ![](media/image1.wmf)                            |
+-------------------+--------------------------------------------------+
| 7.  Review the    | -   Use the appropriate version of the Criteria  |
|     Criteria      |     for Performance Excellence                   |
|     requirements  |     (Business/Nonprofit, Education, or Health    |
|     for your      |     Care) for the application you received.      |
|     assigned      |                                                  |
|     Item.         |                                                  |
+-------------------+--------------------------------------------------+
| 8.  Select and    | -   Considering the Criteria requirements and    |
|     record the    |     all team members' Independent Review         |
|     most relevant |     Worksheets, determine the most relevant four |
|     four to six   |     to six key factors.                          |
|     key factors,  |                                                  |
|     reviewing all | -   Record these key factors at the top of page  |
|     Independent   |     2 of the Consensus Review Item Worksheet.    |
|     Review        |     Use the language in the Consensus Key        |
|     Worksheets    |     Factors Worksheet to ensure consistency. You |
|     for this Item |     may choose to include only a portion of a    |
|     and the draft |     key factor to ensure the most relevance      |
|     consensus Key |     (e.g., one strategic challenge as opposed to |
|     Factors       |     the entire list).                            |
|     Worksheet to  |                                                  |
|     help in your  | > *The key factors, considered in relation to    |
|                   | > the key Criteria requirements, should help     |
|    determination. | > guide your expectations in evaluating the      |
|                   | > applicant's response.*                         |
+-------------------+--------------------------------------------------+
| 9.  Analyze the   | -   Review all the observations on the various   |
|     various       |     Independent Review Worksheets. Note which    |
|     Independent   |     observations the Examiners bolded,           |
|     Review        |     signifying particular importance. Place each |
|     Worksheets,   |     of the Examiners' initials after their       |
|     and begin to  |     observations.                                |
|     determine     |                                                  |
|     commonalities | > *Even though Examiners may categorize the      |
|     and note      | > processes differently, you may find            |
|     conflicts.    | > commonality in the evidence section of the     |
|                   | > worksheet.*                                    |
|                   | >                                                |
|                   | > *You may find it helpful to begin recording    |
|                   | > common ideas on the rationale portion of the   |
|                   | > worksheet, a separate piece of paper, or a     |
|                   | > blank Independent Review Worksheet; or use     |
|                   | > color-coding to organize and sort the key      |
|                   | > approaches and observations. Also consider     |
|                   | > where it is appropriate to give the applicant  |
|                   | > the benefit of doubt.*                         |
|                   |                                                  |
|                   | -   In addition to considering each approach or  |
|                   |     result across all the evaluation factors     |
|                   |     (i.e., horizontally), consider whether you   |
|                   |     might gain overall insights by viewing the   |
|                   |     evaluation factors vertically (e.g., lack of |
|                   |     deployment to volunteers across this Item,   |
|                   |     learning achieved through cycles of          |
|                   |     refinement in more than one process).        |
|                   |                                                  |
|                   | > *This step may require a few iterations for    |
|                   | > you to gain clarity on the most important      |
|                   | > ideas.*                                        |
+-------------------+--------------------------------------------------+
| 10. Use the       | -   Determine the most important core ideas,     |
|     rationale     |     limiting the number to around six total      |
|     portion of    |     strengths and OFIs, and list these in the    |
|     the Consensus |     "Summary of Proposed ..." sections of the    |
|     Item          |     Item Worksheet. List the Item reference(s)   |
|     Worksheet to  |     along with each strength and OFI, recording  |
|     synthesize    |     which Examiners concurred with them.         |
|     about six     |                                                  |
|     most relevant | > *For a sample rationale sheet, see the         |
|     and important | > Consensus Toolkit in the Examiner Learning     |
|     strengths and | > Resource Center                                |
|     opportunities | > (<http://www.baldrige.nist.gov/Word_files/20   |
|     for           | 08_Consensus_Scorebook_Training_Draft_v1.doc>).* |
|     improvement   |                                                  |
|     (OFIs).       | -   In the third "Summary" section of the        |
|                   |     rationale sheet, record any **key**          |
|                   |     conflicts that exist among Examiners.        |
|                   |     (Please avoid listing every observation not  |
|                   |     used.) Propose a resolution to the           |
|                   |     conflicts, and justify why you resolved it   |
|                   |     as you did.                                  |
+-------------------+--------------------------------------------------+
| 11. Develop a     | > *For now, skip the Proposed Scoring Range and  |
|     full comment  | > Score section of the worksheet. You will fill  |
|     for each      | > in this section during step 13.*               |
|     strength and  |                                                  |
|     OFI, linking  | -   In the Strengths and Opportunities for       |
|     it to one or  |     Improvement section on page 2 of the         |
|     more of the   |     worksheet, record a full comment for each    |
|     key factors   |     strength and OFI. Part of your role in       |
|     selected.     |     constructing comments is to create value for |
|                   |     the applicant. This requires more than       |
|                   |     merely restating the applicant's responses.  |
|                   |                                                  |
|                   |     -   For strengths, it involves showing the   |
|                   |         applicant which of its approaches the    |
|                   |         team found to be particularly responsive |
|                   |         to the demands of both the Criteria and  |
|                   |         the organization's operating             |
|                   |         environment, and how the maturity of     |
|                   |         these approaches contributes to          |
|                   |         alignment and improvement.               |
|                   |                                                  |
|                   |     -   For OFIs, it involves showing how the    |
|                   |         lack of an approach or an insufficient   |
|                   |         approach, given Criteria and             |
|                   |         organizational demands, represents a     |
|                   |         potential vulnerability or priority for  |
|                   |         improvement.                             |
+-------------------+--------------------------------------------------+
| 12. Finalize your | -   Download the Comment Guidelines from the     |
|     comments      |     Consensus Toolkit in the Examiner Learning   |
|     using the     |     Resource Center ([www.baldrige.nist.gov/     |
|     Comment       |     Examiner_Resources.htm](http://              |
|     Guidelines.   | www.baldrige.nist.gov/ Examiner_Resources.htm)). |
+-------------------+--------------------------------------------------+
| 13. Using the     | -   Review the comments for the Item, noting     |
|     scoring grid, |     their balance and importance relative to the |
|     determine and |     Item requirements and key factors.           |
|     record a      |                                                  |
|     scoring range | -   Using the first line of the scoring grid in  |
|     and score.    |     the Evaluation Factor Score Summary (labeled |
|                   |     "Approach" for Process Items and "Levels"    |
|                   |     for Results Items), identify which scoring   |
|                   |     range is most descriptive of the             |
|                   |     organization's achievement level. Once you   |
|                   |     have selected a range, read the descriptions |
|                   |     to the left and to the right of it to ensure |
|                   |     that you have selected the most appropriate  |
|                   |     one. Place an "x" in the space below the     |
|                   |     description of the range you selected.       |
|                   |                                                  |
|                   | -   Record the range and a rationale at the      |
|                   |     bottom of the rationale sheet.               |
|                   |                                                  |
|                   | -   Go back to the Consensus Review Item         |
|                   |     Worksheet, and, in turn, review and score on |
|                   |     the scoring grid each of the remaining       |
|                   |     evaluation factors of deployment, learning,  |
|                   |     and integration (for Process Items) or       |
|                   |     trends, comparisons, and integration (for    |
|                   |     Results Items). Record the scoring ranges    |
|                   |     and rationales at the bottom of the          |
|                   |     rationale sheet.                             |
|                   |                                                  |
|                   | ```{=html}                                       |
|                   | <!-- -->                                         |
|                   | ```                                              |
|                   | -   Considering the applicant's performance      |
|                   |     holistically across all evaluation factors,  |
|                   |     determine the overall scoring range for the  |
|                   |     Item. Place an "x" in the box next to the    |
|                   |     range listed in the lower left portion of    |
|                   |     the score summary page.                      |
|                   |                                                  |
|                   | > Some Examiners find it helpful to refer to the |
|                   | > Scoring Guidelines in the appropriate Criteria |
|                   | > booklet to determine the appropriate range.    |
|                   |                                                  |
|                   | -   Determine a percentage score that is a       |
|                   |     multiple of 5 for the Item, and record it in |
|                   |     the blank to the right of the range. Reading |
|                   |     the descriptions of the ranges above and     |
|                   |     below the one you have selected helps in     |
|                   |     determining where the applicant's score      |
|                   |     falls within that range.                     |
|                   |                                                  |
|                   | > *Remember that the applicant does not need to  |
|                   | > demonstrate all characteristics in the         |
|                   | > selected range; rather, the score is based on  |
|                   | > a holistic view of the four evaluation factors |
|                   | > in aggregate.*                                 |
|                   |                                                  |
|                   | -   Record the proposed percentage score at the  |
|                   |     bottom of the rationale sheet.               |
+-------------------+--------------------------------------------------+
| 14. Follow the    | -   Follow the process for virtual consensus on  |
|     instructions  |     the next page, using the file-naming         |
|     for virtual   |     conventions listed in the second column. Be  |
|     consensus.    |     sure to e-mail team members when you have    |
|                   |     completed the virtual consensus steps.       |
+-------------------+--------------------------------------------------+

+-----+------+---------------------------------------+---+---+----------+
| *   |      |                                       |   |   |          |
| *St |      |                                       |   |   |          |
| eps |      |                                       |   |   |          |
| and |      |                                       |   |   |          |
| Fi  |      |                                       |   |   |          |
| le- |      |                                       |   |   |          |
| Nam |      |                                       |   |   |          |
| ing |      |                                       |   |   |          |
| Co  |      |                                       |   |   |          |
| nve |      |                                       |   |   |          |
| nti |      |                                       |   |   |          |
| ons |      |                                       |   |   |          |
| for |      |                                       |   |   |          |
| V   |      |                                       |   |   |          |
| irt |      |                                       |   |   |          |
| ual |      |                                       |   |   |          |
| Co  |      |                                       |   |   |          |
| nse |      |                                       |   |   |          |
| nsu |      |                                       |   |   |          |
| s** |      |                                       |   |   |          |
+-----+------+---------------------------------------+---+---+----------+
| **  | *    | **Steps**                             | * |   |          |
| Ver | *Fil |                                       | * |   |          |
| sio | e-** |                                       | T |   |          |
| n** |      |                                       | e |   |          |
|     | **Na |                                       | a |   |          |
|     | ming |                                       | m |   |          |
|     | C    |                                       | M |   |          |
|     | onve |                                       | e |   |          |
|     | ntio |                                       | m |   |          |
|     | ns** |                                       | b |   |          |
|     |      |                                       | e |   |          |
|     |      |                                       | r |   |          |
|     |      |                                       | I |   |          |
|     |      |                                       | n |   |          |
|     |      |                                       | v |   |          |
|     |      |                                       | o |   |          |
|     |      |                                       | l |   |          |
|     |      |                                       | v |   |          |
|     |      |                                       | e |   |          |
|     |      |                                       | m |   |          |
|     |      |                                       | e |   |          |
|     |      |                                       | n |   |          |
|     |      |                                       | t |   |          |
|     |      |                                       | * |   |          |
|     |      |                                       | * |   |          |
+-----+------+---------------------------------------+---+---+----------+
|     |      |                                       | * | * | **Team   |
|     |      |                                       | * | * | Members  |
|     |      |                                       | I | I | (TMs)**  |
|     |      |                                       | t | t |          |
|     |      |                                       | e | e |          |
|     |      |                                       | m | m |          |
|     |      |                                       | * | B |          |
|     |      |                                       | * | a |          |
|     |      |                                       |   | c |          |
|     |      |                                       | * | k |          |
|     |      |                                       | * | u |          |
|     |      |                                       | L | p |          |
|     |      |                                       | e | \ |          |
|     |      |                                       | a | ( |          |
|     |      |                                       | d | I |          |
|     |      |                                       | * | B |          |
|     |      |                                       | * | U |          |
|     |      |                                       |   | ) |          |
|     |      |                                       | * | * |          |
|     |      |                                       | * | * |          |
|     |      |                                       | ( |   |          |
|     |      |                                       | I |   |          |
|     |      |                                       | L |   |          |
|     |      |                                       | ) |   |          |
|     |      |                                       | * |   |          |
|     |      |                                       | * |   |          |
+-----+------+---------------------------------------+---+---+----------+
| V   | 1    | Item lead:                            | I |   |          |
| ers | _1v1 |                                       | L |   |          |
| ion | (1_  | -   Prepare a first draft of the      | I |   |          |
| 1   | 2v1, |     Consensus Review Item Worksheet,  | B |   |          |
|     | e    |     synthesizing the Independent      | U |   |          |
|     | tc.) |     Review Worksheets from all team   | T |   |          |
|     |      |     members.                          | M |   |          |
|     |      |                                       | T |   |          |
|     |      | -   Post version 1.0 of the worksheet | M |   |          |
|     |      |     (i.e., rationale, Item Worksheet, | T |   |          |
|     |      |     scoring matrix) to the            | M |   |          |
|     |      |     *examinerdepot* Consensus Tab, in |   |   |          |
|     |      |     the appropriate category folder.  |   |   |          |
|     |      |                                       |   |   |          |
|     |      | -   Send an e-mail to the Item backup |   |   |          |
|     |      |     and Team Leader, letting them     |   |   |          |
|     |      |     know it is ready for review.      |   |   |          |
+-----+------+---------------------------------------+---+---+----------+
| V   | 1    | -   Item backup: Post feedback to     | I |   |          |
| ers | _1v2 |     Item lead (file name 1_1v1BU).    | L |   |          |
| ion |      |                                       | I |   |          |
| 2   |      | -   Item lead: Integrate changes as   | B |   |          |
|     |      |     appropriate. Post version 2.0 to  | U |   |          |
|     |      |     *examinerdepot,* and send an      | T |   |          |
|     |      |     e-mail to all team members        | M |   |          |
|     |      |     stating that the version is ready | T |   |          |
|     |      |     for review.                       | M |   |          |
|     |      |                                       | T |   |          |
|     |      | > *The lead and backup may discuss    | M |   |          |
|     |      | > the Item on the telephone, but the  |   |   |          |
|     |      | > Team Leader must be notified that   |   |   |          |
|     |      | > the step is complete.*              |   |   |          |
+-----+------+---------------------------------------+---+---+----------+
| Dr  | KTd  | -   Team Leader or designated team    | ( |   |          |
| aft | raft |     member: Use Consensus Review Item | K |   |          |
| Key |      |     Worksheets to develop an initial  | T |   |          |
| The |      |     list of key theme (KT) ideas.     | l |   |          |
| mes |      |     Post a draft, and send an e-mail  | e |   |          |
|     |      |     to all team members stating that  | a |   |          |
|     |      |     the draft version is ready for    | d |   |          |
|     |      |     review.                           | o |   |          |
|     |      |                                       | n |   |          |
|     |      |                                       | l |   |          |
|     |      |                                       | y |   |          |
|     |      |                                       | ) |   |          |
|     |      |                                       | T |   |          |
|     |      |                                       | M |   |          |
|     |      |                                       | T |   |          |
|     |      |                                       | M |   |          |
|     |      |                                       | T |   |          |
|     |      |                                       | M |   |          |
+-----+------+---------------------------------------+---+---+----------+
| T   | Use  | Team members:                         | I |   |          |
| eam | Ex   |                                       | L |   |          |
| Mem | amin | -   Review version 2.0 of each        | I |   |          |
| ber | er's |     Consensus Review Item Worksheet   | B |   |          |
| Fe  | i    |     and the draft KTs.                | U |   |          |
| edb | niti |                                       | T |   |          |
| ack | als, | -   Provide feedback to each Item     | M |   |          |
|     | e    |     lead by posting comments to the   | T |   |          |
|     | .g., |     appropriate folder on             | M |   |          |
|     |      |     *examinerdepot*.                  | T |   |          |
|     | 1_1v |                                       | M |   |          |
|     | 2_MS | > *Depending on the nature of the     |   |   |          |
|     |      | > feedback, you may use colored text  |   |   |          |
|     | 1_1v | > on the actual Item Worksheet to     |   |   |          |
|     | 2_BF | > show the proposed changes.*         |   |   |          |
|     |      | >                                     |   |   |          |
|     | 1_1v | > *You must provide feedback for      |   |   |          |
|     | 2_PH | > every Consensus Review Item         |   |   |          |
|     |      | > Worksheet and the draft KTs by      |   |   |          |
|     | K    | > posting to* examinerdepot, *even if |   |   |          |
|     | Tdra | > you simply post a document that     |   |   |          |
|     | ftMS | > says "no changes."*                 |   |   |          |
|     |      |                                       |   |   |          |
|     | K    |                                       |   |   |          |
|     | Tdra |                                       |   |   |          |
|     | ftBF |                                       |   |   |          |
|     |      |                                       |   |   |          |
|     | K    |                                       |   |   |          |
|     | Tdra |                                       |   |   |          |
|     | ftPH |                                       |   |   |          |
+-----+------+---------------------------------------+---+---+----------+
| V   | 1    | Item leads and KT lead:               | I |   |          |
| ers | _1v3 |                                       | L |   |          |
| ion |      | -   Integrate feedback from the final | I |   |          |
| 3   | KT   |     review of Version 2.0 or the      | B |   |          |
|     | call |     draft KTs, and draft the version  | U |   |          |
|     |      |     you are proposing for final       | T |   |          |
|     |      |     consideration during the          | M |   |          |
|     |      |     consensus call. If you make major | T |   |          |
|     |      |     changes, adjust the rationale to  | M |   |          |
|     |      |     describe these changes.           | T |   |          |
|     |      |                                       | M |   |          |
|     |      | -   Post version 3.0 for Items and    |   |   |          |
|     |      |     KTcall for key themes to          |   |   |          |
|     |      |     *examinerdepot* along with        |   |   |          |
|     |      |     revised rationales, as            |   |   |          |
|     |      |     appropriate.                      |   |   |          |
|     |      |                                       |   |   |          |
|     |      | -   Send an e-mail to all team        |   |   |          |
|     |      |     members letting them know that    |   |   |          |
|     |      |     changes have been made and new    |   |   |          |
|     |      |     versions are ready for review.    |   |   |          |
|     |      |                                       |   |   |          |
|     |      | Team members:                         |   |   |          |
|     |      |                                       |   |   |          |
|     |      | -   Review all Version 3.0 and KT     |   |   |          |
|     |      |     documents in preparation for the  |   |   |          |
|     |      |     consensus call.                   |   |   |          |
|     |      |                                       |   |   |          |
|     |      | ```{=html}                            |   |   |          |
|     |      | <!-- -->                              |   |   |          |
|     |      | ```                                   |   |   |          |
|     |      | -   If you as an Item lead receive    |   |   |          |
|     |      |     conflicting input or you as a     |   |   |          |
|     |      |     team member still have concerns   |   |   |          |
|     |      |     about particular comments or      |   |   |          |
|     |      |     scoring issues, send an e-mail to |   |   |          |
|     |      |     the Team Leader and the team,     |   |   |          |
|     |      |     requesting a discussion during    |   |   |          |
|     |      |     the consensus call.               |   |   |          |
|     |      |                                       |   |   |          |
|     |      | -   If you agree with the content of  |   |   |          |
|     |      |     all Consensus Review Item         |   |   |          |
|     |      |     Worksheets, send an e-mail to the |   |   |          |
|     |      |     Team Leader and team indicating   |   |   |          |
|     |      |     that you have completed the       |   |   |          |
|     |      |     review and have no Items to       |   |   |          |
|     |      |     recommend for discussion.         |   |   |          |
+-----+------+---------------------------------------+---+---+----------+
| Po  | 1    | -   Item lead and KT lead: If the     | I |   |          |
| stc | _1_C |     team agrees on changes, integrate | L |   |          |
| all | onse |     them and post the consensus       | I |   |          |
| Con | nsus |     version to *examinerdepot* for    | B |   |          |
| sen |      |     inclusion in the Consensus        | U |   |          |
| sus | KT   |     Scorebook.                        | T |   |          |
| V   | C    |                                       | M |   |          |
| ers | onse |                                       | T |   |          |
| ion | nsus |                                       | M |   |          |
|     |      |                                       | T |   |          |
|     |      |                                       | M |   |          |
+-----+------+---------------------------------------+---+---+----------+

+----------------------+-----------------------------------------------+
| **STEPS**            | **ACTIONS**                                   |
+======================+===============================================+
| 15. Team Leader:     | Sample agenda:                                |
|     Draft the        |                                               |
|     consensus call   | -   Welcome                                   |
|     agenda,          |                                               |
|     including only   | -   Agenda review                             |
|     Items or         |                                               |
|     comments that    | -   Round-robin process for the team to use   |
|     require          |     in capturing inputs                       |
|     discussion, and  |                                               |
|     post the agenda  | -   2^nd^ and 3^rd^ strengths for 1.1; impact |
|     on               |     on scoring                                |
|     *examinerdepot*. |                                               |
|                      | -   Items 5.1 and 5.2; scoring for 5.1 and    |
|                      |     5.2                                       |
|                      |                                               |
|                      | -   Item 7.4                                  |
|                      |                                               |
|                      | -   Key themes                                |
|                      |                                               |
|                      | -   Item and overall scoring review           |
|                      |                                               |
|                      | -   Scoring band discussion                   |
|                      |                                               |
|                      | -   Next steps                                |
+----------------------+-----------------------------------------------+
| 16. Item leads:      | -   If the entire Item is to be discussed,    |
|     Prepare to       |     prepare to review the Criteria            |
|     present          |     requirements, key factors, and            |
|     pertinent Items  |     clarification of any areas of             |
|     or comments.     |     disagreement to ensure that the team is   |
|                      |     prepared to discuss the Item.             |
|                      |                                               |
|                      | -   If only one comment is to be discussed,   |
|                      |     prepare to discuss the Item requirement   |
|                      |     in question and any pertinent key         |
|                      |     factors. A discussion on the Item scoring |
|                      |     should follow.                            |
+----------------------+-----------------------------------------------+
| 17. Participate in   | -   Team members: Be prepared to serve in     |
|     the consensus    |     additional assigned roles (e.g., Criteria |
|     call: The team   |     cop, scorekeeper). Also be prepared to    |
|     covers a limited |     provide specific points and               |
|     number of Items  |     recommendations for changes. Throughout   |
|     or comments,     |     the discussion, listen for changes that   |
|     reviews and      |     may impact or conflict with other Item-   |
|     updates the Key  |     or key-theme-level comments, and bring    |
|     Theme and the    |     them to the attention of the team. **Note |
|     Score Summary    |     that ALL team members must be present to  |
|     Worksheets, and  |     proceed with the call.**                  |
|     determines the   |                                               |
|     Process and      | -   Item leads: Ensure that changes in your   |
|     Results scoring  |     Consensus Review Item Worksheets are      |
|     bands.           |     correctly reflected in the Key Themes     |
|                      |     Worksheet and that scores are correctly   |
|                      |     recorded on the Score Summary Worksheet.  |
|                      |                                               |
|                      | -   Item backups: Take notes for Item leads.  |
|                      |     It may be helpful for you to restate what |
|                      |     you hear at the end of each topic.        |
|                      |                                               |
|                      | -   ***New step for 2008:*** Determine and    |
|                      |     discuss the scoring bands for Process and |
|                      |     Results Items, using the subtotal scores  |
|                      |     on the Score Summary Worksheet and the    |
|                      |     Scoring Band Descriptors. If the band     |
|                      |     descriptors are misaligned with point     |
|                      |     scores,                                   |
|                      |                                               |
|                      | ```{=html}                                    |
|                      | <!-- -->                                      |
|                      | ```                                           |
|                      | -   adjust the band if the score is on the    |
|                      |     margin of two bands, or                   |
|                      |                                               |
|                      | -   review the Item scores and adjust as      |
|                      |     appropriate---giving consideration to all |
|                      |     comments, including comments that have    |
|                      |     not been discussed.                       |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
| 18. Review scoring.  | -   Scorekeeper: Review what you recorded for |
|                      |     each Item.                                |
|                      |                                               |
|                      | -   Item leads: Confirm that there is         |
|                      |     agreement on each Item score and that the |
|                      |     score is recorded correctly on individual |
|                      |     Item Worksheets and the Score Summary     |
|                      |     Worksheet.                                |
+----------------------+-----------------------------------------------+
| 19. Post final       | -   Scorekeeper: At the end of the call, post |
|     scores.          |     the Score Summary Worksheet on            |
|                      |     *examinerdepot,* and inform the Team      |
|                      |     Leader and ASQ (telephone 1-414-765-7205) |
|                      |     that it is posted.                        |
+----------------------+-----------------------------------------------+
| 20. Item and key     | -   Item backups: Review the Item lead's      |
|     theme leads:     |     Consensus Review Item Worksheet to help   |
|     Revise comments  |     ensure that the revised comments and      |
|     to reflect       |     scores accurately represent the consensus |
|     consensus        |     discussion.                               |
|     decisions.       |                                               |
|                      | -   Team Leader: Review the key themes to     |
|                      |     ensure that all changes or updates have   |
|                      |     been made.                                |
+----------------------+-----------------------------------------------+
| 21. Post final       | -   Item leads: Post the final worksheets on  |
|     worksheets by    |     *examinerdepot*.                          |
|     the              |                                               |
|     preestablished   | -   Team Leader: Assemble the final Consensus |
|     deadline.        |     Scorebook, conduct a final check, and     |
|                      |     inform NIST that the final scorebook      |
|                      |     (vFinal) is posted.                       |
+----------------------+-----------------------------------------------+