Criteria Based Course Evaluation Process

For the purpose of reviewing prospective providers of any criteria-based
training product, it is an agency responsibility to establish a standard
process to evaluate the course delivery and materials as presented by
the provider's cadre. The NWCG Leadership Committee role is to provide
course evaluation assistance to any agency upon request.

### 

### EVALUATION TEAM

The evaluation process should be conducted using a team that consists of
2 or 3 individuals, with a team leader appointed. All members of the
evaluation team must have strong involvement in the area of training at
the geographic area or national level. They must have previously
attended or helped deliver the particular course in question. Each of
the evaluators must have instructor experience at the 300 level or
higher in the NWCG curriculum.

**[EVALUATION FORMAT]{.underline}**

Courses evaluated must meet NWCG leadership curriculum intent, cadre
requirements, instructional design guidelines, and address all content
objectives as outlined in the L-381 Course Criteria. This information
should be identified in an evaluation form. The NWCG Leadership
Committee has developed a suggested evaluation form on the following
pages of this document and a suggested format to summarize the
evaluation team observations as follows:

[General Findings]{.underline}

Deficiencies and Suggestions:

The final written evaluation shall focus on the areas in need of
improvement (termed deficiencies) as observed by the evaluation team.
These should be agreed upon by the evaluation team. Observations from
the evaluation team should provide specific details and examples of the
key areas in need of improvement. There should also be suggestions that
are shared with the provider cadre to help insure improvement.

Summary and Recommendation:

The final written evaluation should give specific recommendations to the
reviewing agency and to the provider as to future actions related to
this course. If the course is approved, it should be stated clearly how
the agency(s) should proceed in using the course provider. If the course
is not approved, it should be stated how the agency(s) can continue to
work with the course provider in order to achieve an approved course, or
if they should discontinue working with the course provider in the
future. Names of agency contacts where the course provider can address
questions should be listed.

**[RECORDS]{.underline}**

A copy of the evaluation package and related course materials reviewed
should be filed and kept for future reference.

# **Evaluator Profile**

Name: \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Job Title:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Home Unit:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Phone #:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Email:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Experience as a fire instructor (check highest level of experience):

  ------------------------------------------------------------- ---------
  100/200 level fire courses                                    

  300 level fire courses                                        

  400 level fire courses                                        

  500/600 level fire courses                                    
  ------------------------------------------------------------- ---------

Experience with Leadership curriculum (check all that apply):

  ----------------------------------- ---------------- ------------------
  **Course**                          **Attended**     **Instructed
                                                       and/or
                                                       Coordinated**

  L-180 Human Factors on the Fireline                  

  L-280 Followership to Leadership                     

  L-380 Fireline Leadership                            

  L-381 Incident Leadership                            
  ----------------------------------- ---------------- ------------------

## Course Information

Location:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Date:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Names of cadre being presenting course (include company or fire agency):

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

+-----------------------+-----------------------------------+----------+
| **ELEMENT**           | **GUIDELINE**                     | **       |
|                       |                                   | RATING** |
+-----------------------+-----------------------------------+----------+
|                       |                                   |          |
+-----------------------+-----------------------------------+----------+
| # **Cadre**           |                                   |          |
+-----------------------+-----------------------------------+----------+
| Cadre experience      | Combined skill set of the cadre   | Yes / No |
|                       | includes expert level knowledge   |          |
|                       | in principle-centered leadership  |          |
|                       | training, emergency incident      |          |
|                       | operations, and human factors     |          |
|                       | research.                         |          |
+-----------------------+-----------------------------------+----------+
| Cadre commitment      | -   Cadre members share           | Yes / No |
|                       |     instructor tasks and provide  |          |
|                       |     a good variety of             | Yes / No |
|                       |     presentations for students    |          |
|                       |     throughout the course.        | Yes / No |
|                       |                                   |          |
|                       | -   More than one cadre member is |          |
|                       |     usually present in the        |          |
|                       |     classroom at any given time.  |          |
|                       |                                   |          |
|                       | -   Cadre members make themselves |          |
|                       |     available for individual      |          |
|                       |     student questions             |          |
|                       |     before/after sessions and at  |          |
|                       |     breaks.                       |          |
+-----------------------+-----------------------------------+----------+
| Instructor            | -   All cadre members are         | Yes / No |
| qualifications        |     facilitative instructor       |          |
|                       |     qualified:                    | Yes / No |
|                       |                                   |          |
|                       |     -   Attended and passed the   | Yes / No |
|                       |         M-410 course or           |          |
|                       |         equivalent.               |          |
|                       |                                   |          |
|                       |     -   Demonstrated ability to   |          |
|                       |         facilitate simulations,   |          |
|                       |         role-play and other group |          |
|                       |         exercises.                |          |
|                       |                                   |          |
|                       | -   Lead instructor has           |          |
|                       |     demonstrated experience in    |          |
|                       |     this same role presenting     |          |
|                       |     complex multi-day courses.    |          |
+-----------------------+-----------------------------------+----------+
| Instructor            | All instructors were credible and | Yes / No |
| credibility           | demonstrated respect for the      |          |
|                       | students and the sponsor agencies |          |
|                       | within the wildland fire service. |          |
+-----------------------+-----------------------------------+----------+
| Instructor            | All instructors are prepared      | Yes / No |
| preparation           | with:                             |          |
|                       |                                   | Yes / No |
|                       | -   Well-written and appropriate  |          |
|                       |     student materials.            | Yes / No |
|                       |                                   |          |
|                       | -   Well-designed and appropriate |          |
|                       |     audio/visual aids.            |          |
|                       |                                   |          |
|                       | -   Presentations that promote    |          |
|                       |     student participation and     |          |
|                       |     kept podium based lecture to  |          |
|                       |     a minimum.                    |          |
+-----------------------+-----------------------------------+----------+

+-----------------------+-----------------------------------+----------+
| **ELEMENT**           | **GUIDELINE**                     | **       |
|                       |                                   | RATING** |
+-----------------------+-----------------------------------+----------+
|                       |                                   |          |
+-----------------------+-----------------------------------+----------+
| # **I                 |                                   |          |
| nstructional Design** |                                   |          |
+-----------------------+-----------------------------------+----------+
| Wildland Fire         | -   The prework reading           | Yes / No |
| Leadership            |     assignment:                   |          |
| Development Program   |                                   | Yes / No |
| components            |     -   Title was taken from the  | Yes / No |
|                       |         Professional Reading      |          |
| Refer to:             |         Program.                  | Yes / No |
|                       |                                   |          |
| w                     |     -   All students had the      | Yes / No |
| ww.fireleadership.gov |         book.                     |          |
|                       |                                   | Yes / No |
|                       |     -   Examples from the reading |          |
|                       |         assignment were utilized  | Yes / No |
|                       |         throughout the course.    |          |
|                       |                                   |          |
|                       | -   The Values and Principles     |          |
|                       |     model was used and reinforced |          |
|                       |     throughout the course.        |          |
|                       |                                   |          |
|                       | -   Required Leadership Toolbox   |          |
|                       |     items were referenced:        |          |
|                       |                                   |          |
|                       |     -   Briefing & Intent Guide   |          |
|                       |                                   |          |
|                       |     -   After Action Review Guide |          |
|                       |                                   |          |
|                       |     -   Self-development Plan     |          |
+-----------------------+-----------------------------------+----------+
| Training in context   | Group exercises and simulations:  | Yes / No |
|                       |                                   |          |
|                       | -   Comprised at least 50% of the | Yes / No |
|                       |     course presentation time.     |          |
|                       |                                   | Yes / No |
|                       | -   Simulations were the primary  |          |
|                       |     training technique utilized   |          |
|                       |     throughout the course.        |          |
|                       |                                   |          |
|                       | -   Exercises and simulations are |          |
|                       |     related the target audience's |          |
|                       |     level of leadership           |          |
|                       |     responsibility.               |          |
+-----------------------+-----------------------------------+----------+
| Case studies          | Case study examples were          | Yes / No |
|                       | frequently utilized from other    |          |
|                       | high-risk work environments.      |          |
+-----------------------+-----------------------------------+----------+
| After Action Review   | The After Action Review process   | Yes / No |
|                       | was consistently integrated into  |          |
|                       | group exercises and simulations   |          |
|                       | as a performance assessment and   |          |
|                       | improvement tool.                 |          |
+-----------------------+-----------------------------------+----------+

+-----------------------+-----------------------------------+----------+
| Integrated curriculum | Course content was consistent     | Yes / No |
|                       | with content and concepts from    |          |
|                       | the preceding three courses in    |          |
|                       | the NWCG Leadership Curriculum    |          |
|                       | (L-180 *Human Factors,*           |          |
|                       |                                   |          |
|                       | L-280 *Followership to            |          |
|                       | Leadership,* and L-380 *Fireline  |          |
|                       | Leadership*).                     |          |
+-----------------------+-----------------------------------+----------+
| Time                  | Course delivery designed to       | Yes / No |
|                       | provide at least 32 contact hours |          |
|                       | between instructors and students. |          |
+-----------------------+-----------------------------------+----------+

+----------------------+------------------------------------+----------+
| **ELEMENT**          | **GUIDELINE**                      | **       |
|                      |                                    | RATING** |
+----------------------+------------------------------------+----------+
|                      |                                    |          |
+----------------------+------------------------------------+----------+
| # **Content**        |                                    |          |
+----------------------+------------------------------------+----------+
| Command Presence and | The course content effectively     | Yes / No |
| Climate              | addresses the following            |          |
|                      | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Describe techniques and        |          |
|                      |     processes that contribute to   |          |
|                      |     building a positive command    |          |
|                      |     climate.                       |          |
|                      |                                    |          |
|                      | 2.  Establish a positive command   |          |
|                      |     climate that promotes          |          |
|                      |     communication and error        |          |
|                      |     mitigation within a command    |          |
|                      |     team and with assigned         |          |
|                      |     operational resources.         |          |
+----------------------+------------------------------------+----------+
| Leaders Intent       | The course content effectively     | Yes / No |
|                      | addresses the following            |          |
|                      | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Provide clear command guidance | Yes / No |
|                      |     and initial leaders intent for |          |
|                      |     an emerging incident.          |          |
|                      |                                    |          |
|                      | 2.  Communicate and clarify        |          |
|                      |     command guidance and leaders   |          |
|                      |     intent to subordinates         |          |
|                      |     regarding changing environment |          |
|                      |     and changing mission           |          |
|                      |     objectives.                    |          |
|                      |                                    |          |
|                      | 3.  Evaluate the effectiveness of  |          |
|                      |     leaders intent and subordinate |          |
|                      |     understanding from an incident |          |
|                      |     and develop future trigger     |          |
|                      |     points for detecting errors.   |          |
+----------------------+------------------------------------+----------+

+----------------------+------------------------------------+----------+
| Command and Control  | The course content effectively     | Yes / No |
|                      | addresses the following            |          |
|                      | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Exercise appropriate levels of |          |
|                      |     command so that leaders intent |          |
|                      |     is clear and positive control  |          |
|                      |     is maintained as distance      |          |
|                      |     increases between the command  |          |
|                      |     team and assigned operational  |          |
|                      |     resources.                     |          |
|                      |                                    |          |
|                      | 2.  Provide a command climate that |          |
|                      |     encourages cooperation and     |          |
|                      |     communication with clients     |          |
|                      |     (such as public,               |          |
|                      |     municipalities, law            |          |
|                      |     enforcement, political         |          |
|                      |     leaders, etc.)                 |          |
+----------------------+------------------------------------+----------+
| Rapid Team Building  | The course content effectively     | Yes / No |
|                      | addresses the following            |          |
|                      | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Describe the functions of a    | Yes / No |
|                      |     high performing team and the   |          |
|                      |     techniques for rapid           |          |
|                      |     teambuilding in a              |          |
|                      |     time-constrained environment.  |          |
|                      |                                    |          |
|                      | 2.  Apply the teambuilding         |          |
|                      |     principles to achieve cohesion |          |
|                      |     in a quickly assembled command |          |
|                      |     team.                          |          |
|                      |                                    |          |
|                      | 3.  Identify typical team/crew     |          |
|                      |     stress profiles and develop    |          |
|                      |     stress control measures for    |          |
|                      |     common stress reactions within |          |
|                      |     a team/crew.                   |          |
+----------------------+------------------------------------+----------+

+----------------------+------------------------------------+----------+
| Communication        | The course content effectively     | Yes / No |
| Techniques           | addresses the following            |          |
|                      | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Demonstrate communication      | Yes / No |
|                      |     techniques that focus the      |          |
|                      |     organization\'s efforts and    |          |
|                      |     minimize conflict and error in |          |
|                      |     high-stress, high-consequence  |          |
|                      |     situations. As a minimum, the  |          |
|                      |     techniques addressed should    |          |
|                      |     include the five               |          |
|                      |     communications                 |          |
|                      |     responsibilities for           |          |
|                      |     firefighters and active        |          |
|                      |     listening.                     |          |
|                      |                                    |          |
|                      | 2.  Demonstrate adapting           |          |
|                      |     communication techniques to    |          |
|                      |     the unique needs of different  |          |
|                      |     people or situations.          |          |
|                      |                                    |          |
|                      | 3.  Participate in the After       |          |
|                      |     Action Review process to       |          |
|                      |     provide opportunities to learn |          |
|                      |     as a team from a previous      |          |
|                      |     event.                         |          |
+----------------------+------------------------------------+----------+
| Detecting and        | The course content effectively     | Yes / No |
| Mitigating           | addresses the following            |          |
| Operational Error    | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Describe error detection and   | Yes / No |
|                      |     mitigation processes at the    |          |
|                      |     command level.                 | Yes / No |
|                      |                                    |          |
|                      | 2.  Integrate error and            |          |
|                      |     contingency planning into an   |          |
|                      |     ongoing operation, and         |          |
|                      |     identify and mitigate error    |          |
|                      |     chains as they occur.          |          |
|                      |                                    |          |
|                      | 3.  Enable subordinates to         |          |
|                      |     participate in error detection |          |
|                      |     and mitigation processes       |          |
|                      |     during an ongoing operation.   |          |
|                      |                                    |          |
|                      | 4.  Anticipate upstream or         |          |
|                      |     systemic errors that may       |          |
|                      |     impact your team during an     |          |
|                      |     ongoing operation.             |          |
+----------------------+------------------------------------+----------+

+----------------------+------------------------------------+----------+
| Operational Tempo    | The course content effectively     | Yes / No |
| Awareness            | addresses the following            |          |
|                      | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Describe factors that affect   | Yes / No |
|                      |     operational tempo and identify |          |
|                      |     risks and challenges           |          |
|                      |     associated with them.          |          |
|                      |                                    |          |
|                      | 2.  Detect shifts in operational   |          |
|                      |     tempo and span of control      |          |
|                      |     within an ongoing operation.   |          |
|                      |     Adapt tactics, plans, and      |          |
|                      |     leadership techniques to       |          |
|                      |     mitigate risks.                |          |
|                      |                                    |          |
|                      | 3.  Analyze the effectiveness of a |          |
|                      |     team/crew\'s detection and     |          |
|                      |     management of changes in       |          |
|                      |     operational tempo on an        |          |
|                      |     incident and develop future    |          |
|                      |     trigger points for managing    |          |
|                      |     those changes.                 |          |
+----------------------+------------------------------------+----------+
| Self-development     | The course content effectively     | Yes / No |
| Actions              | addresses the following            |          |
|                      | objectives:                        | Yes / No |
|                      |                                    |          |
|                      | 1.  Conduct a self-assessment of   | Yes / No |
|                      |     individual leadership skill    |          |
|                      |     weaknesses/strengths and       |          |
|                      |     review previous leadership     |          |
|                      |     development plans.             |          |
|                      |                                    |          |
|                      | 2.  Construct a development plan   |          |
|                      |     to enhance personal leadership |          |
|                      |     skills using peer feedback,    |          |
|                      |     self-assessment of             |          |
|                      |     performance, and instructor    |          |
|                      |     feedback.                      |          |
|                      |                                    |          |
|                      | 3.  Analyze the effects of         |          |
|                      |     personal stress reactions on   |          |
|                      |     decision-making. The analysis  |          |
|                      |     should include identification  |          |
|                      |     of stressors in a scenario and |          |
|                      |     determination of how stress    |          |
|                      |     reactions may have affected    |          |
|                      |     ability to make decisions.     |          |
+----------------------+------------------------------------+----------+

## General Findings

Deficiencies and Suggestions:

Summary and Recommendation:

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Signature Date