2010 NCSL International Workshop and Symposium 

Converting Technical Content to Training Material 

Speakers/Authors: Georgia L. Harris, Dana Leaman 

National Institute of Standards and Technology (NIST) 

100 Bureau Drive 

Gaithersburg, MD 20899 

G. Harris: 301-975-4014, gharris@nist.gov

D. Leaman: 301-975-4679, dana.leaman@nist.gov

Abstract: This paper provides the basis for an interactive mini tutorial and covers how to 

convert technical content to training materials. It includes: defining the audience, writing 

Learning Objectives, designing content and activities to achieve objectives, engaging participants 

in learning activities using adult learning methods, and assessing the learning event to determine 

whether objectives have been met. Examples are provided from NCSL International1 resources. 

The instructional approach in this paper covers the Analysis, Design, and Development phases of 

the ADDIE instructional system development (ISD) model; due to time constraints, it will only 

briefly touch on the Implementation and Evaluation phases. The paper integrates concepts from 

Bloom’s Taxonomy and criteria from the ANSI/IACET2

 standard for offering continuing 

education units as an Authorized Provider. 

Learning Objectives: Given the handouts and practical experience during the tutorial session, 

participants will be able to successfully: 

1. Identify the phases of the ADDIE instructional design model; 

2. Define the appropriate audience for training content; 

3. Identify and Create well-written Learning Objectives; 

4. Give examples of Activities that will engage adult participants and achieve Learning 

Objectives; and 

5. Identify appropriate Assessment methods to determine whether Learning Objectives have 

been met. 

Background 

The NCSLI Strategic Plan has identified an effort to 

create training resources to match with NCSLI 

publications as they are created and updated. NCSLI 

is also seeking to gain compliance with the 

International Association for Continuing Education 

and Training (ANSI/IACET) criteria for offering 

Authorized Provider Continuing Education Units 

(CEUs) to ensure continual improvement and 

professional training approaches in our metrology 

training. This session will use resources from three 

                                                            

1

 NCSL International (also known as National Conference of Standards Laboratories, International). 

2

 American National Standards Institute, International Association for Continuing Education and Training 

(ANSI/IACET). http://www.iacet.org, 2010. 

Figure 1. Sample ANSI/IACET logo which we 

could apply to NCSLI content if we were an 

Authorized Provider.
2010 NCSL International Workshop and Symposium 

new NCSLI publications and their associated conference/section meeting presentations as 

Application examples throughout this paper. These publications include Recommended Practice 

(RP) 3, “Calibration Procedures”, Recommended Practice (RP) 20, “Metrology Laboratory 

Workforce Planning” and the “Metrology Human Resource Handbook” (HR Handbook). 

The ANSI/IACET criteria require organizations to use a systematic design and development 

process for developing all training materials. Most professional instructional designers follow 

some type of model, probably the most common of which is the ADDIE instructional system 

development (ISD) process. ADDIE refers to the Analysis, Design, Development, 

Implementation, and Evaluation phases of the ISD process. One of the unique features of the 

metrology community is that instructors are often subject matter experts (SMEs) without a 

formal background in ISD. Our overall goal with this paper and mini-tutorial is to provide 

guidance to the NCSLI technical committee members who are SMEs to develop training 

resources that follow a standardized practice, or model process, that will enable consistency in 

course development as well as compliance to the ANSI/IACET requirements. 

We will cover two modules (with integrated Activities) during the mini-tutorial. The first module 

provides background information on the ADDIE instructional system development model and 

the other provides Application examples for each phase in the ADDIE model. However, this 

paper integrates the Applications with each topic as it is covered. We have also organized the 

mini-tutorial modules directly around the five Learning Objectives stated on the first page. We 

will reflect on the Learning Objectives as we cover this ISD model and as we apply the model to 

our three case studies. 

One of the key things to consider in all adult learning events is that adults often preview your 

objectives or abstract to determine if there is something in your session for them. They ask the 

question “what’s in it for me?” Adults juggle many priorities and their time is valuable. So, we 

can apply that concept, right now – why are you here? What is it in this particular session that 

you hope to get? What’s in it for you? Why are you reading this paper? During the mini-tutorial, 

we will take some time to reflect on what aspects of the session are most important for each 

participant. 

To keep this session most applicable, we have selected three case studies and are relying on the 

subject matter experts (SMEs) from the committees responsible for the content to help us ensure 

the content meets the needs of the participants who might receive training. So, our design 

process overlays these educational design concepts onto the technical content. We hope that this 

approach will serve as a useful model for speakers who want to ensure that ideas they present are 

applied in the workplace and that committee members who want to develop training material 

based on the technical content in guides, standards, and recommended practices are able to 

follow these steps to be successful in creating effective training materials. Our number one goal 

in training materials is to be able to reach a designated level of knowledge or application. 
2010 NCSL International Workshop and Symposium 

ADDIE Instructional Design Model 

There are many instructional system development models and you can see it graphically 

presented in a number of ways. The Laboratory Metrology Group of the Weights and Measures 

Division, National Institute of Standards and Technology (NIST) have chosen this particular 

approach because it follows a Plan, Do, Check, Act (PDCA) model with Evaluation forming a 

part of every phase. We will work through this diagram (See Fig. 2) and each of the phases, 

starting with Analysis. Every instructional designer ends up tailoring this model to his own 

processes, projects, and approaches to developing training. “One cardinal rule is to never leave 

out Analysis or Evaluation from the learning 

event development process because the 

projects can be spotted quickly – 1) these 

efforts seldom work to meet learning 

objectives and 2) no one ever really figures 

out why.”3

. 

There are a number of websites that cover 

Instructional Design concepts and the ADDIE 

model. Some additional references include: 

 Hodell, Chuck, “ISD From the Ground 

Up, A No-Nonsense Approach to 

Instructional Design,” ASTD Press, 

August 2006. 

 Elengold, Linda J., “Teach SMEs to 

Design Training,” ASTD Info-Line, 

Tips, Tools, and Intelligence for 

Trainers, 2001. 

 Harris, Georgia L., “Development of a 

CD-ROM Metrology Course at 

NIST,” Measurement Science 

Conference, 2003. It is available here: 

http://ts.nist.gov/WeightsAndMeasures/upload/MSC2003CDROMPaper.pdf

Defining the Audience and Need for Training 

During the Analysis phase, the designer or, in most metrology cases, the instructor or SME 

defines the need, the target audience, and the expected outcome of the training. ANSI/IACET 

Criteria Number 4 is related to Learning Event Planning (4): “Each learning event is planned in 

response to the identified needs of a target audience.” There is room on the Case Study Planning 

Worksheet (Appendix A) and Learning Event Planning Worksheet (Appendix B) to make notes 

about the Audience and Need for a given training event. 

As a first step in Analysis, we need to answer a number of questions: 

 Who is the audience? 

                                                            

3 Hodell, Chuck, “ISD From the Ground Up, A No-Nonsense Approach to Instructional Design,” ASTD Press, 

August 2006. 

Figure 2. ADDIE Instructional System Development 

(ISD) Model. ADDIE: Analysis, Design, Development, 

Implementation, Evaluation.
2010 NCSL International Workshop and Symposium 

 Why conduct the training? 

 What is the performance need? 

 What is the root cause? 

 How will the content be delivered and by whom? 

We may need to answer these questions from the perspective of laboratory management as well 

as the metrologist or person being trained. We must have an effective partnership between the 

trainer, the manager, and the person being trained for the training to be effective and 

used/applied back on the job. If a manager does not support change that might be required for 

applying training content, the training will have no impact. Identifying the real need and the best 

solution are important for everyone. 

One thing we might want to ask is: can other solutions meet the requirements without training? 

Sometimes when analyzing the need, we find that the root cause is lack of management support 

or lack of resources, rather than a lack of knowledge or awareness. Perhaps a simple publication 

and job aid such as a form or checklist serves an even better purpose than spending time in a 

training session. A step by step checklist or form may ensure consistent application of a new 

procedure, publication, or idea, without need for a training course. 

If we look at the ANSI/IACET criteria for offering CEUs, one of the things we find is that we 

can also refer to a “job standard” to define the need for training. For example, any single item in 

the ISO/IEC 17025 standard for calibration and testing laboratories is rich as training content in 

the calibration world. One example might be a course on “Writing a Calibration Report 

(ISO/IEC 17025, Section 5.10.” 

Application: Identifying the Audience for our NCSLI Publications 

- The audience for the mini tutorial is primarily Committee Chairs and Members who 

want to develop training material from NCSLI publications. There might be 

additional benefits to regular conference presenters or tutorial instructors who want to 

improve the instructional value of their resources. 

- The audience for RP 3, “Calibration Procedures” might be: Calibration Laboratory 

Managers, Metrologists/Engineers, Technical Managers, or the Procedure 

Writing/Validation Team. 

- The audience for RP 20, “Metrology Laboratory Workforce Planning” and the 

“Metrology Human Resources Handbook” might include: Laboratory Managers, 

Human Resources staff, and Training Managers/Directors. 

Application: Examples of Need 

- The need for this mini-tutorial is that SMEs need formal training on methodologies 

and processes for developing training material from NCSLI publications to ensure 

compliance with standard training methodologies and the ANSI/IACET requirements. 

- The need for training on RP 3, “Calibration Procedures” could include: a 

requirement in ISO/IEC 17025 to document calibration procedures and validate them. 

- The need for training on RP 20, “Metrology Laboratory Workforce Planning” and the 

“Metrology Human Resources Handbook” include: a desire for international 
2010 NCSL International Workshop and Symposium 

consistency and adoption of standardized job descriptions to enable recognition and 

professional status of metrology careers. 

Learning Objectives 

A Learning Objective or Learning Outcome (often interchangeably used), is a specific statement, 

written from the participant’s perspective, which provides information about what the participant 

will gain during a learning event. They are focused on participant performance, not teacher 

performance. 

“Learning objectives: Statements about what a student will gain from a course or activity. These 

are specific statements about exactly what a student should know, be able to do, or value as a 

result of accomplishing a learning goal. Learning objectives form the basis for curriculum and 

course development as well as testing (Reed, 2005).”4

 

The “Bloom’s to Assessment” graphic (Fig. 5) and the Learning Event Planning Worksheet 

(Appendix B) are two tools that will help implement these concepts. They will help answer 

“what” and “why” of our learning event. Part of the Analysis phase helps determine what level 

of training and comprehension is required by the audience. Then, the Design process requires 

that we design training at the level needed to help the participant get what they need at the right 

level. 

Bloom’s Taxonomy 

We will consider these 

six levels of 

understanding before 

we consider writing 

effective Learning 

Objectives. We need 

to answer what level 

we want the 

participant to be able 

to know and apply the 

material. We must 

accurately identify the 

audience, understand 

their level of 

knowledge, and their 

unique needs. Each of 

the six areas in the 

taxonomy builds on 

the previous level of 

knowledge. A key 

design and 

                                                            

4

From the University of Texas at Dallas, glossary: http://sacs.utdallas.edu/sacs_glossary (March 2010). 

Figure 3. Bloom's Taxonomy Model.
2010 NCSL International Workshop and Symposium 

development concept is that a participant must have Knowledge about a topic before they can 

Analyze it. In Table 1, the six areas noted in the Bloom’s graphic (Fig. 3) are listed, with a brief 

description of each category, and a list of verbs that can be used to describe what the participant 

will need to be able to Know, Do, or Think after the session. These sample verbs provide 

examples that can be used to reflect appropriate learning levels in each Learning Objective and 

to specify the level of mastery expected for the student. 

Table 1. Bloom's Taxonomy - Descriptions and Sample Verbs. 

Level Description Sample Verbs 

Knowledge Recall data or information describe, identify, recall, arrange, define, duplicate, 

label, list, memorize, name, order, recognize, 

reproduce state 

Comprehension Understand the meaning of a 

problem, be able to translate into 

own words 

comprehend, give example, classify, describe, 

discuss, explain, express, identify, indicate, locate, 

recognize, report, restate, review, select, translate 

Application Use a concept in a new situation apply, change, construct, compute, choose, 

demonstrate, dramatize, employ, illustrate, interpret, 

operate, practice, schedule, sketch, solve, use, write 

Analysis Can split concepts into parts and 

understands the structure 

analyze, break down, relate, appraise, calculate, 

categorize, compare, contrast, criticize, 

differentiate, discriminate, distinguish, examine, 

experiment, question, make inferences, find 

evidence, test 

Synthesis Produce something from different 

elements (e.g., a report) 

summarize, arrange, combine, categorize, assemble, 

collect, compose, construct, create, design, develop, 

formulate, manage, organize, plan, prepare, 

propose, set up, write 

Evaluation Make judgments, justify a 

solution 

appraise, interpret, argue, assess, attach, compare, 

defend, estimate, judge, predict, rate, core, select, 

support, value, evaluate, prove, deduct 

How to Write Learning Objectives 

The ANSI/IACET standard for continuing education units identifies four categories in Section 5 

related to writing Learning Objectives. These are the four criteria required for writing an 

effective Learning Objective. 

1. They are written from the perspective of the learner, reflecting what the learner will achieve. 

2. Learning objectives must be clear, specific, concise, and measurable (with four components): 

a. They state the performance the learner should be able to accomplish. (Behavior) 

b. They specify the conditions under which the learner is to perform. (Conditions) 

c. They specify the criteria for acceptable performance. (Criteria) 

d. They are directly related to the subject matter and content of the learning event. 
2010 NCSL International Workshop and Symposium 

3. Learning outcomes are established for each session within a large event, conference, or 

convention. 

4. Instructional delivery includes discussion of learning outcomes. 

If we expand on category number 2, and consider the four components of a clear, specific, 

concise, and measurable objective, here are some additional notes to clarify what is meant. Each 

Learning Objective should begin with: After this session (tutorial, paper, or workshop) the 

participant will__________. 

Component 1: This component covers the expected behavior after the training. Think about 

performance in terms of active verbs related to what you want the participant to know, do, or be, 

after the training: identify, calculate, assess, present, analyze, and apply. (Refer back to Table 1 

for additional examples.) At this point, select an appropriate verb for the level of knowledge or 

application that is expected. 

Component 2: What are the conditions? Can the participant use their notes? Can they use a 

documented procedure? Can they use a calculator? Are computers allowed? Must they use Excel 

for calculations? Are there additional reference materials provided? Will they have to be 

assessed from memory? 

Component 3: What criteria will be used to judge acceptable performance? Is an 80 % passing 

grade acceptable? Would it be okay if they submit their response in text-message format? Must 

they provide a written response or can it be oral? What will a valid uncertainty statement look 

like? (Instructors need to make sure that the criteria for successful performance are covered in 

the course!) 

Component 4: Learning objectives must be directly related to the subject matter and content of 

the event. If you haven’t covered various types of statistical distributions in a course, you should 

not evaluate students against the criteria (unless of course it was given as a prerequisite). If a 

course is to cover how to correctly perform pressure calibrations, it would not make sense to 

have Learning Objectives related to the laboratory management system. This component should 

be obvious – but must be stated. 

Another approach commonly used is the A-B-C-D approach to writing Learning Objectives. A, 

B, C, and D stand for Audience, Behavior, Condition, and Degree. This approach matches up 

nicely with the ANSI/IACET criteria, in that the objective must focus on the Audience, and be 

written from the student perspective. Then, it needs to specify what Behavior is expected as a 

result of the training, must address the Conditions that will be allowed, and the Degree or level 

of mastery required (the Criteria for measuring successful mastery). We may not use this model 

in this tutorial, but you might see it in some references on this topic and the approach may be 

helpful to you. 

Developing Learning Objectives for our Application examples are next. You can see that we 

have included the Behavior, Condition, and Criteria in these examples. Note how you might 

improve or expand on ideas for appropriate Learning Objectives. 
2010 NCSL International Workshop and Symposium 

Application: Examples for NCSLI Publications 

- Five Learning Objectives for this mini tutorial were stated earlier on page 1. 

- Objectives for RP 3, “Calibration Procedures” might include: given resources and 

examples (condition), participants will be able to correctly (criteria) write, assess 

(identify good procedures, identify gaps and weaknesses), and validate procedures 

(behavior). 

- Objectives for RP 20, “Metrology Laboratory Workforce Planning” might include: 

given the resources (condition), participants will be able to describe the overall 

workforce planning process (behavior), and successfully implement all or portions of 

(criteria) laboratory succession planning efforts (behavior), etc. 

- Objectives for the “Metrology Human Resources Handbook” might include: given the 

resources (condition), participants will be able to update job descriptions (behavior) 

consistent with standard practice (criteria), collect employment data (behavior) 

according to standard classifications (criteria), participate in providing input to 

OPM/Department of Labor, etc. 

How to Select and Align Activities and Assessments with Learning Objectives 

The triangle shown in 

Figure 4 represents the 

relationship between 

Learning Objectives, 

Learning Activities and 

Assessment. If these three 

components are present 

and compatible then 

teaching and learning is 

enhanced, hence, this 

model is often called “The 

Magic Triangle.” If these 

three components are not 

congruent then students 

become discouraged and 

unhappy and make the 

assumption the objectives 

cannot be trusted and they 

will stop paying attention 

to them. A key factor to consider with this model is that if one side of the triangle is missing, the 

learning collapses and is not effective. 

Note: Learning Activities are those things the instructional designer plans during the Design 

Phase and the student does to learn in the Implementation Phase. For example, listening to a 

lecture is a Learning Activity; engaging in a small group discussion led by a facilitator is a 

Learning Activity; evaluating a measurement instrument with a calibration technician clinician is 

a Learning Activity. 

Figure 4. "The Magic Triangle." Used to align Learning Objectives, 

Activities, and Assessment Methods.
2010 NCSL International Workshop and Symposium 

Evaluation or Assessment (of the student, not the course) is often thought of as a testing

component. But, Assessment could also be a project assignment that is graded or otherwise 

evaluated. The important factor to consider is that whatever forms the Assessment takes, it 

should measure the student’s accomplishment and provide specific feedback to the student(s) on 

how well they met the Learning Objectives. 

Figure 5. From Bloom's to Assessment. 

If you review the “Bloom’s to Assessment” graphic (Fig. 5), you can see that there are some 

activities more appropriate for some levels of learning than others. E.g., a Lecture/Test might be 

appropriate Activity and Assessment for the Knowledge level learning, but it is usually the 

lowest level of engagement and retention. A Case Study Activity and Assessment are more 

appropriate for the Application and Synthesis levels. The concept of the Magic Triangle (Fig. 4) 
2010 NCSL International Workshop and Symposium 

should be considered when using the Learning Event Planning Worksheet (Appendix B) that we 

will also discuss later. Keep this idea in mind: the Objective, Activity, and Assessment 

components are all a part of the triangle, all are essential, and all must be considered during the 

Design process. They must be selected to match the appropriate level of Bloom’s taxonomy. 

Designing Content 

ANSI/IACET Criteria 7 states that: the content and instructional methods are appropriate for 

each learning outcome; content is organized in a logical manner in support of learning outcomes; 

instructional methods are consistent with learning outcomes regardless of delivery mode; and 

instructional methods accommodate various learning styles and are designed to promote 

interaction between and among learners, instructors, and learning resources to achieve the stated 

learning outcomes. 

Depending on what topic is being converted to training material, a trainer might design around 

themes, chronology, or steps in a measurement process. It is important to make sure that topics 

are aligned around the objectives and to focus on ensuring that the technical content is effective 

as training material. Using a logical sequence of topics is one of the ANSI/IACET criteria and is 

important for training course development. One topic should typically build on the knowledge 

gained from previous topics or modules. 

In this mini tutorial we are taking PowerPoint®5

 slide content that is “about a publication” and 

converting it to a “training resource.” If the content were not available, we would have to start 

from scratch and design and build everything. Our focus is to convert the content in such a way 

as to comply with recognized ISD education and training models. 

Techniques, teaching methods, or activities need to be selected and aligned for each of our Case 

Studies to match the Bloom’s Taxonomy level we want to achieve, as well as the KSA 

(Knowledge, Skill, Ability) we are trying to ensure the participant can KNOW or DO at the 

conclusion of the session. What are the best instructional methods that are likely to be used 

during the Implementation phases? Instructional designers must think about best instructional 

and Assessment methods during Analysis and Design to select the best activities and methods for 

teaching. They must also consider the best Assessment methods. Table 2 lists several examples 

of Teaching Activities/Methods. 

A traditional model of training that combines Activity and Assessment is Lecture, followed by a 

Test. In a conference setting, we often only see Lectures. As you can see by reviewing the 

“Bloom’s to Assessment” graphic (Fig. 5), this approach is not very effective if you want 

participants to know or do something different. What makes it worse is that adults prefer to be 

involved in their learning and tend to hate the Lecture/Test model. What comes to mind is “death 

by PowerPoint!” This approach provides the lowest level of engagement and retention and treats 

the audience as inexperienced/non experts (though they usually do bring something to the 

learning event). During the Analysis and Design phase we must answer whether we want people 

to only be able to LIST information or be able to fully ANALYZE and APPLY the material on 

                                                            

5

 The use of specific software products is not intended as an endorsement; it is simply the products that are 

commonly used in the development of training resources. 
2010 NCSL International Workshop and Symposium 

the job. In most cases, we want to see performance improvements on the job and not simply 

improve a participant’s knowledge. 

Table 2. Teaching Activities/Methods. 

Activity or Method Description 

Lecture Presents factual material in a direct logical method. Useful for large groups. Should only 

be used in combination with other instructional techniques, unless there is a very 

inspirational instructor and the goal is to inspire learners. Lectures must include liberal 

use of visual, auditory, and kinesthetic (physical activity) approaches. Lectures are often 

passive and learning is generally at the lowest end of Bloom’s taxonomy. 

Video Can be entertaining if selected and done well. Video can be used as part of an Activity, 

but additional instructions need to be included, such as “watch this video and take notes 

on the measurement errors you observe.” 

Discussion Discussions are often started by asking open ended questions and engaging all 

participants. They can be done in large groups, small groups, or as brainstorming 

sessions. The idea is to engage all participants. It is not practical for very large groups 

unless there is an option for breaking into smaller groups and reporting back. 

Demonstration In this method, the instructor or a member of the class might demonstrate a procedure or 

method to the rest of the class. Participants are given an idea of what something might 

look like in practice and have the opportunity to observe, ask questions, and critique. 

Exercise/Drill Perform a measurement procedure or review a specific procedure is an example of an 

exercise/drill activity. Exercises provide immediate opportunities to apply what has been 

learned. Students individually perform an experiment and then come together as a class 

to discuss the results. The hands-on application of a procedure and final expected results 

are often assessed. 

Case Study Case studies are practical examples related to the content that must also be relevant to 

the participant. Allows for application of content that has been learned and promotes 

analytical and problem solving skills. The case study must be complete enough for the 

participant to assess the entire case versus having too many unknown components that 

might simply frustrate the students (especially if the right answer is more fixed than 

students are led to believe). 

Application: Selecting Activities 

To convert our technical content to training materials, we need to have some creative planning. 

Having a team of designers or instructors select energetic and applicable Activities can be an art. 

This phase is often quite a challenge for no-nonsense SMEs for whom the content is obvious and 

comes easily. During the design process we need to answer: What kind of activities will help 

participants understand, implement, analyze at the levels needed? We have selected some 

Activities for our Case Studies below. Can you identify additional Learning Activities that might 

be fun, engaging, and effective? 

- Activities for this mini tutorial. We are making extensive use of real case studies from 

new NCSLI publications. These provide real examples of what we are trying to do. 
2010 NCSL International Workshop and Symposium 

- Activities for RP 3, “Calibration Procedures” might include: provide a procedure 

with parts missing – identify missing parts and consider the impact if they are not 

present; provide a procedure that is poorly written or unclear and consider the impact; 

provide a well written procedure, and consider what is needed to document the 

validation. 

- An Activity for RP 20, “Metrology Laboratory Workforce Planning” might include: 

perform a knowledge/skills/ability Assessment for participants (make it real for each 

person) to answer the question, what would be required for succession planning to fill 

your own position. 

- An Activity for the “Metrology Human Resources Handbook” might include: bring 

your own set of job descriptions to the course to assess them against the Handbook 

criteria. 

How to Ensure Learning Takes Place–Assessments 

Activities and Assessment methodologies must be considered in the context of Design, but the 

instructional designer must consider how they will be Implemented for each audience. The 

ANSI/IACET Criteria 8 specifies that Assessment must take place. Procedures established 

during event planning [Design, Development] are used to assess student’s achievement of the 

learning outcomes. Learners must also be provided feedback on their mastery of learning 

outcomes. During development we will develop the instructional content, but we must consider 

good design approaches in the Design phase. Later, Implementation also includes the Activities, 

and Assessments of whether students have mastered the topic at the expected or desired level. 

It is important to align the Assessment method with the Learning Objective and Activity as 

shown in the Magic Triangle (Fig. 4) and the “Bloom’s to Assessment” graphic (Fig. 5). For 

adult audiences, it is important to integrate Assessment into the training as much as possible 

(versus issuing a post-test). According to the ANSI/IACET criteria, each person does not have to 

be assessed on achieving each objective in a course, but sometimes that is important for issuing 

certificates of successful completion or for demonstration of competency. For example, if you 

have objectives such as “each person will successfully calculate the standard deviation and get 

100 % correct” or “each person will successfully calibrate item x during the seminar.” In those 

examples, the instructor will need to be able to review each person’s numbers or measured result 

to assess accuracy and achievement of the objectives, and also to provide feedback to the 

student. Sometimes there is also a need to assess post-event learning and application – e.g., use 

of proficiency testing for procedure training. 

To improve Assessment efficiency, and to motivate and engage students, it is important to 

provide group feedback to let them know what you learned from the Assessments and what 

difference that information will make. All Assessments need to reflect back to measurable 

Learning Objectives to determine if the student has learned the material and it is important to 

provide feedback to the student so that they know whether they have learned the material 

correctly. That is, Assessment should be more than simply correcting a quiz and returning it with 

a grade. Table 3 provides a number of Assessment Methods that may be considered. 
2010 NCSL International Workshop and Symposium 

Table 3. Assessment Methods. 

Assessment Method Description 

Multiple Choice A type of quiz or Assessment where the student must choose either the best answer or 

a number of answers that might be correct. Immediate review can be done to provide 

feedback. Discussion of “wrong” items often yields interesting applications that might 

not have been considered during the Analysis, Design, and Development phases. 

True False This is a simple type of quiz or Assessment that can be done orally or on paper. It can 

be done individually or in a large group setting. Right and wrong answers are usually 

provided. It can be done by raising of hands. E.g., how many of you think this 

statement is True? How many false? Can anyone give me a reason for their answer? 

Feedback might be as simple as “great job” or “here’s the correct answer.” 

Fill in Blanks This type of approach can be done in a group setting and handled orally, or even in 

teams. Most often it is used in a written test and will usually only have one right 

answer. 

Essay/Reports It is more difficult to assess essays/reports in a training session unless it is several days 

long and the instructor will have time to review and grade the content. Specific 

guidance may be needed in the Developed material to ensure that an instructor or 

facilitator knows the content adequately or has adequate information on which to base 

evaluation. A “one minute” Assessment can also be useful. For example, “write down 

one thing you learned [and can apply] during this session on this index card and pass it 

in.” 

Simulation The Assessment is made in each participant’s engagement, completion, and accuracy 

of the Learning Objectives and at the level of knowledge required. Job task analysis 

can help create an observational checklist. 

Case Study The application of a procedure and final expected results are often assessed. Group 

review of the final results can provide dynamic feedback or individual feedback may 

be provided to each participant. Teachers can use a checklist and observation to assess 

student success with the particular material. A minimum set of knowledge or skills can 

be included on the checklist. Job task analysis can help create an observational 

checklist. 

Role Play Specific guidance is given to the parties in the role play and then observations and 

feedback from the group are collected. The instructor provides an overview of the 

Activity and summarizes the important components. Participants are encouraged to 

practice a role or skill. Teachers can use a checklist and observation to assess student 

success with the particular material. 

Journaling Students are asked to take specific notes during the session. For example, “write down 

one key idea from this session and write down one thing you can apply back on the 

job.” The summary notes are reviewed at the end of the class either as a group or 

individually. The notes can also be used to summarize or highlight the important 

aspects of what was learned to a manager or coworker back on the job. 
2010 NCSL International Workshop and Symposium 

The Case Study Worksheet (Appendix A) and Learning Event Planning Worksheet (Appendix 

B) use tables that can be expanded like the one shown in Table 4. These worksheets can be using 

during the Design phase to consider effective methods in training resource development. But, 

this specific portion of the worksheets (Table 4) helps align the Learning Outcomes with the 

Activities and Assessment methods to ensure that all three sides of the Magic Triangle (Fig. 4) 

are aligned according to the “Bloom’s to Assessment” (Fig. 5) and that the instructional 

materials and methods will be effective. If Assessments are integrated into the Activity 

effectively, students may not realize whether they are learning or being assessed. 

Table 4. Example Learning Outcomes, Instructional Methods, and Assessment Methods. 

Item Learning Outcome Instructional Method 

(Activity) 

Assessment Method 

 

Application: Matching Assessments to Activities and Learning Objectives 

- Activities and Assessment for this mini tutorial. We are making use of real case 

studies and reviewing possible Applications as we cover each Phase. Assessment will 

be done by evaluating the levels of participation and questions raised by participants. 

Feedback will be given during the session. Follow up Assessment will be done to 

determine the level at which new training materials comply with the ADDIE model 

and ANSI/IACET criteria. 

- Activities for RP 3, “Calibration Procedures” might include: provide a procedure 

with parts missing – identify missing parts and consider the impact if they are not 

present; provide a procedure that is poorly written or unclear and consider the impact; 

provide a well written procedure, and consider what is needed to document the 

validation. In this case, was the participant able to identify missing items in a 

calibration procedure? This could be done through group discussions and a 

presentation with feedback from the facilitator. Assessing impact would be much 

more difficult as an Activity and one might question whether there is a list of key 

items one might consider in assessing the impact of poorly written calibration 

procedures. The content of the Assessment material must be covered in the training. 

- An Activity for RP 20, “Metrology Laboratory Workforce Planning” might include: 

perform a knowledge/skills/abilities Assessment for participants (make it real for each 

person) to answer the question, “What would be required for succession planning to 

fill your own position?” If an Activity included discussion of each phase in 

Succession Planning, and the Learning Objective was to list three of the five phases, 

an oral examination, or team competition, might be used to assess whether the 

participants understood the topic at the level expected. 

- An Activity for the “Metrology Human Resources Handbook” might include: bring 

your own set of job descriptions to the course to assess them against the Handbook 

criteria. However, the objective might have been to be able to list the three job titles 
2010 NCSL International Workshop and Symposium 

that are in the handbook. In that case, a quiz at the end could determine if the 

participant is able to list all three job descriptions. A matching quiz could be used to 

match attributes with each job title. 

Developing a Lesson Plan, Agenda, Instructor’s Resources 

Once the course content is Designed, the course materials are Developed. Content that is 

developed might include an Agenda, Lesson Plan, Slides, Visual Aids, and Instructors Notes. It 

should also include any additional handouts or case study references that are not included as a 

part of the course reference materials. For example, in this mini-tutorial, we provide: a copy of 

this paper, copies of PowerPoint slides from the three Application examples, and additional 

References on a CD-ROM. We are also providing a set of PowerPoint slides (Notes version) and 

a paper copy of Appendix A for personal notes during the session. We are intentionally 

providing the Notes version because our time will be limited and we want to encourage 

participants to share this resource within their committees or with others who are developing 

training resources. Our goal is to encourage participants to implement the content at the 

Application level of Bloom’s Taxonomy. You will note that this goal was not stated as a 

Learning Objective, but is an outcome of the training that we hope will be fulfilled. 

The agenda for the mini-tutorial includes: 

- Overview of the Session (including Learning Objectives). We will ask participants to 

take notes on their own project or use the Case Study we will discuss. 

- Module 1: ADDIE ISD Process/Theory. This is presented as Lecture with Group 

Oral Assessment. 

- Module 2: ADDIE Process – Applied to One Case Study (the RP on Calibration 

Procedures). (Note: this paper provides references to three Applications, but there is 

not time to cover all three examples during the session; however they are included in 

the Notes section of the slides/handouts.) The Five Steps of Adult Learning 

(Metrologist, July 2010) will be implemented as a part of the teaching/Activity. 

Assessment will include review of participant engagement. Are the questions 

applicable? Are groups interacting? Are new, creative ideas raised? 

For a mini-tutorial that is presented in one and a half hours, we are not planning for any breaks. 

However, we want to cover the Overview and Module 1 fairly quickly so that we can ensure we 

spend the majority of our time on the Applications (Case Study). An estimated time might be 15 

minutes for an Introduction and Overview, 30 minutes for Module 1, and 45 minutes for Module 

2. 

In general, PowerPoint® provides a Notes section that can be used for instructor guidance and 

reference. During the Development, and Implementation phases, it is important to ensure that the 

notes and content are enough for a knowledgeable and skilled instructor or SME to present the 

content. What else might be needed? Are the case studies included? Will there be a compiled 

handout for participants as well as instructors? What qualifications are needed (if any) for the 

instructor/facilitator? The instructor must have a background in the instructional design process, 

but because the process is fairly simple and the notes fairly complete, a good instructor could 

feasibly review the materials and facilitate the implementation with knowledgeable SMEs. In 
2010 NCSL International Workshop and Symposium 

fact, the content for this mini-tutorial could easily be shared by an effective Committee Chair to 

the rest of their Committee to guide the development of new training resources. 

Implementation and Evaluation 

Implementation and Evaluation are the last two phases of the ADDIE model. It is arguable that 

Evaluation is last because we integrate it into each level of the process. It is beneficial to conduct 

dry-run evaluations and obtain feedback (beta testing) with instructors or SMEs to ensure the 

needs and objectives of the training material will be met. Adjustments in the content will often 

be made as the material is developed – and before the course is presented. Course evaluations 

and continual improvement (refinement) of content, objectives, activities, and Assessments also 

help to improve content over time. 

Having an evaluation program that includes overall assessment of all program components, as 

well as individual course evaluations, is an essential part of the ANSI/IACET criteria, but we 

will not spend time on those phases during the mini-tutorial. This is not to minimize their 

importance as a part of the ADDIE process as much as it is related to the need to focus on 

Designing and Developing content in the available time. Recall what we said earlier: a cardinal 

rule is to never leave out Analysis or Evaluation from the learning event development process 

because the projects can be spotted quickly – 1) these efforts seldom work to meet learning 

objectives and 2) no one ever really figures out why.” 

Conclusions 

This paper is being presented as part of a mini-tutorial to help metrology subject matter experts 

design and develop training content that follows formal ISD models and to help comply with 

ANSI/IACET standards for Authorized Providers in NCSLI-developed training resources. The 

authors hope that this resource will be expanded and enhanced for use by NCSLI committees in 

the Analysis, Design, Development, Implementation, and Evaluation of training resources as 

well as used as a resource for converting or developing effective metrology training content. 

Acknowledgements 

The authors with to acknowledge contributions from the Committee Chairs/Co-chairs 

(committee number provided), Gloria Neely (163), Caroline Dixon (163), and Thomas Flynn 

(176) for providing PowerPoint® slides related to each of their new publications and for 

participating in a preliminary webinar on these topics to provide subject matter guidance, 

Application examples, and feedback on the draft mini-tutorial presentations. 

Appendices and References 

 Appendix A: Case Study Planning Worksheet 

 Appendix B: Learning Event Planning Worksheet 

 Metrologist Articles: CEU (April 2009), Train the Trainer: Writing Learning Objectives 

(April 2010), 5 Steps for Adult Learning (July 2010), Activities and Assessments 

(October 2010, draft) 

 NCSLI Website: Trainer Resources: 

http://www.ncsli.org/NCSL/learning/Trainer_Resources.aspx 
2010 NCSL International Workshop and Symposium 

Appendix A 

Case Study Planning Worksheet 

Title of Course/Tutorial:_________________________________ 

ADDIE Phase Questions and Notes 

Analysis What is the need? 

Who is the audience? 

What standards might be referenced? 

What will success look like? 

Design Write Learning Objectives: 

Using ____________________, participant will be able to _________________ (at this 

level_______________) after the training. 

Level in Bloom’s Taxonomy? 

Possible VERBS? 

Match Activity – Align Assessment Method 

What are some possible activities and Assessments that can be used? 

Learning Objective Activity Assessment 

 

Tools: Learning Event Planning Worksheet, “Bloom’s to Assessment”, Writing Learning Objectives Article 

Develop Creating content... 

Teacher’s Manual. Slides. Case Studies. 

Often slides.... (or outline/notes) 

Additional handouts? Worksheets? Case studies? 

Additional considerations: Instructor notes. Agenda. Lesson Plan. Timing/breaks. Constraints. 

Implement Step 1: Set up the Activity. 

Step 2: Conduct the Activity. 

Step 3: Learners share and interpret their reactions. 

Step 4: Participants identify concepts. 

Step 5: Participants consider and share how they will apply these concepts. 

Tools: Five Steps of Adult Learning 

Evaluate 1. Evaluate each phase of the ADDIE process. 

2. Participant Assessment Methods. 

3. Use of standard course evaluations. 

Tools: Learning Event Planning Worksheet, “Bloom’s to Assessment” 
2010 NCSL International Workshop and Symposium 

Appendix B 

Sample Learning Event Planning Worksheet (used for Analysis and Design Phases) 

Title: 

 

Abstract: 

 

Constraints: 

 Instructor: 

 Time/Date: 

 Length of course: 

 Prerequisites: (e.g., previous course, downloadable reading, tasks/activities) 

 Maximum number of students: 

 Minimum number of students: 

 Room set up: 

 AV Required: 

Define the audience and need for this training (e.g., laboratory management knowledge, skill, ability, standards 

such as ISO/IEC 17025, 17043, VIM, GUM, measurement parameter knowledge, skill, ability to perform 

calibrations, poor performance on proficiency tests, observed/requested needs): 

 
2010 NCSL International Workshop and Symposium 

Complete this table for each course/event: 

Learning Objectives6

 Instructional Method (Activity)7 Assessment Method8

 

Agenda Outline (Include highlight agenda items and attached detailed agenda and descriptions.)

Topic Time 

Attendance, Group Introductions 

Course Title, Description, and Successful Completion requirements 

Learning Objectives 

Module 1 Objective, Activity, Assessment 

Module 2 Objective, Activity, Assessment 

Module 3 Objective, Activity, Assessment 

Review and Closure 

Course and Student Assessments 

                                                            

6 Note Bloom’s Taxonomy and write from the Learner’s perspective... E.g., At the end of this session, the learner 

will be able to use “x” tool to comply with section “y” of “q” standard; will be able to name the benefits of 

complying with the standard; will be able to tell others how to select the best tools from choices given to implement 

a program. This is NOT where the instructor says “here’s what we’re going to cover.” Suitable activities and 

Assessment methods are to be used to ensure participant learning at the desired level. 7

 Think about Instructional methods and Learning activities. What are the Visual activities: observing slides, 

watching list of notes, demonstrations of documents, watching a measurement, watching a video, reading a 

procedure. What are the Auditory cues: lecture, discussion, question/answer, round the room sharing, brainstorming, 

and role playing? What are the kinesthetic activities: making a measurement, taking time to reflect and write notes – 

journaling; sharing thoughts; reviewing and summarizing a document; note taking that includes key questions (and 

can be used for learner Assessment). Activities must be matched with Assessment method and Learning Objectives. 8

 Think about how to provide immediate Assessment and feedback to adult learners and consider the Five Steps of 

Adult Learning. After a question directed to the group (whether they answer aloud or in writing), provide the right 

answer. Use a quick daily/hourly Quiz. Use a Q&A approach and provide answers. Ask individual participants to 

share their answers or examples. Use a competitive game. Use role playing and have participants provide feedback 

to each other. Give immediate feedback when creating a group list by brainstorming.