# Presentation: 218539

## Testing Strategies:  To Test or Not to Test?

- Prepared by the
- caBIGTM Documentation and Training WorkspaceLead Authors: Lara Fournier & Ranjani Ramakrishnan
- Oregon Health & Sciences University (OHSU)

- Criteria for Including Testing with caBIGTM Training

## Overview

- Background
- Determining if Testing is Needed
- A Decision Tool (Risk/Complexity/Feasibility Table)
- Case Scenarios
- Review & References

**Overview** | Determining if Testing is Needed | A Decision Tool | Case Scenarios | Review & Resources

## Background

- What is a test?
- A method to evaluate whether learners can demonstrate abilities and skills related to content being taught. 1
- “A deliberate attempt by people to acquire information about themselves or others.” 2
- Testing may include:
**Qualification Test** – Often conducted _post-training_, this evaluation test measures learner against a benchmark performance level; skills must be demonstrated to allow some future action (e.g., tool access).
**Pre-Test/Test-Out** – Evaluates learner’s abilities _prior_ to training; if scores are high enough, may skip some/all training modules.
**Self-Check Test** – Allows learner to self-check learning as training unfolds. 1

**Overview** | Determining if Testing is Needed | A Decision Tool | Case Scenarios | Review & Resources

## Background

- Cultural perceptions of testing
*“**Two adjectives, good and fair, are the keys to testing.  **Without them, tests become one of the most hated elements of modern living*.”2
- What will the student get out of it?
- If nothing – don’t test
- Developing tests, especially Qualification Level, can be time consuming & expensive
- Students deserve to know why they are taking a test.  Too often this is overlooked.2

**Overview** | Determining if Testing is Needed | A Decision Tool | Case Scenarios | Review & Resources

## Determining if Testing is Needed

- Three Considerations
**Risk**
- The potential for harm if the student does not learn the material
- Primary consideration for Qualification Level testing when  risk is high
**Complexity**
- Multi-faceted dimension that includes course objectives, content volatility, audience, total course length and training type.
- Can guide whether other forms of testing could enhance training (i.e. self-tests) when risk is medium or low.
**Feasibility**
- The resources you have available to develop and administer tests.

- Overview | **Determining if Testing is Needed** | A Decision Tool | Case Scenarios | Review & Resources

## Determining if Testing is Needed 

- Assessing Risk - types to consider:
- Harm to Information Systems
- Disruption, corruption, breach or other types of damage to information systems, databases, data, hardware or software (e.g. source code).
- Risk to Safety
- Harm (physical, emotional, etc.) to a patient, study subject, coworker, or general public
- Potential for Litigation
- Lawsuits, litigation, or regulatory compliance violation (e.g. violations of patient privacy, HIPAA)
- Can often accompany either of the above risks
- Harm to Workflow
- Disruption or damage  to departmental, organizational or regional workflow(s) (e.g. a bottleneck in a production line halts delivery)

- Overview | **Determining if Testing is Needed** | A Decision Tool | Case Scenarios | Review & Resources

## Determining if Testing is Needed

- Assessing Risk - types to consider (continued):
- Harm to Image
- Damage to department or organizational reputation, image or competitive advantage (e.g. losing college accreditation)
- Efficiency
- Loss or degradation of organizational or departmental efficiency (e.g.  Workflow slows and results in longer wait time)
- The student
- Potential for harm to the student themselves (e.g. unable to advance to next level of coursework)

- Overview | **Determining if Testing is Needed** | A Decision Tool | Case Scenarios | Review & Resources

## Assessing Complexity
Course Objectives
From “Introduction to”   “Mastery of”
The more advanced/challenging the objectives, the more likely testing can enhance the training
Volatility of Content
Content that changes frequently
Testing can be a tool to focus student on material that’s changed 
Audience
Training should be developed with your student in mind
Try to anticipate your expected audience type (i.e. Nurses, Programmers, study coordinators, etc...)
Mixed audiences can be more challenging to reach
Risk talking over the heads of some and boring others

- Course Objectives
- From “Introduction to”   “Mastery of”
- The more advanced/challenging the objectives, the more likely testing can enhance the training
- Volatility of Content
- Content that changes frequently
- Testing can be a tool to focus student on material that’s changed
- Audience
- Training should be developed with your student in mind
- Try to anticipate your expected audience type (i.e. Nurses, Programmers, study coordinators, etc...)
- Mixed audiences can be more challenging to reach
- Risk talking over the heads of some and boring others

- Determining if Testing is Needed

- Overview | **Determining if Testing is Needed** | A Decision Tool | Case Scenarios | Review & Resources

## Assessing Complexity (continued)
Course length/total time
If lengthy, then testing can be used to add interest and variety between lectures
Consider in-class, take-home, interactive, or computer-based exercises

Type of Training
The method of delivery for training material should factor into decision.
Tests can add interest and keep students engaged when delivery method lacks real-time, direct interaction with instructor or other students (e.g. Self-Directed).

- Course length/total time
- If lengthy, then testing can be used to add interest and variety between lectures
- Consider in-class, take-home, interactive, or computer-based exercises
- Type of Training
- The method of delivery for training material should factor into decision.
- Tests can add interest and keep students engaged when delivery method lacks real-time, direct interaction with instructor or other students (e.g. Self-Directed).

- Determining if Testing is Needed

- Overview | **Determining if Testing is Needed** | A Decision Tool | Case Scenarios | Review & Resources

## Determining if Testing is Needed

- Assessing Feasibility
- Resources available (feasibility of implementation)
- What tools and staff you have available
- SCORM compliant software tools for developing tests
- Staff available to help develop, administer and update material
- If risk is low and resources are limited then testing makes less sense

- Overview | **Determining if Testing is Needed** | A Decision Tool | Case Scenarios | Review & Resources

## A Decision Tool

- The Risk / Complexity / Feasibility Table
- A 2-D tool to guide in the decision process for those developing training materials
- Not a scientific tool but an exercise
- Risk, Complexity and Feasibility attributes are weighted
- Complete the following three questions sets and sum your totals together
- Find your place in the table based on your score
- Results are a recommendation, not mandatory

- Overview | Determining if Testing is Needed | **A Decision Tool** | Case Scenarios | Review & Resources

## Assessing Risk

- If sum total Risk > 75 = ‘High’
- between 25-75 = ‘Med’
- <25 = ‘Low’

| √ | If student doesn’t learn the material, is there any risk of the following? | Value |
| --- | --- | --- |
|  | Disruption, corruption, breach, or damage to network and/or information systems, databases, data, hardware or software (source code)... | 75 |
|  | Compromise of safety to patient, subject, coworker or general public | 75 |
|  | Potential litigation, lawsuit, liability or regulatory compliance violation | 75 |
|  | Disruption or damage to departmental, organizational, or regional workflow(s) | 50 |
|  | Damage to departmental or organizational reputation, image or competitive advantage | 50 |
|  | Loss or degradation in departmental or organizational efficiency | 15 |
|  | Lack in adherence to non-regulatory standards | 15 |
|  | Loss of student’s own potential to elevate or promote to new position | 5 |
|  | Compromise student’s own ability to advance to next level or coursework | 5 |
|  | Compromise student’s own ability to take on new challenge in this area | 5 |

- Overview | Determining if Testing is Needed | **A Decision Tool** | Case Scenarios | Review & Resources

## Assessing Complexity – Step 1

| √ | Course Objectives: Your training can be classified as...(select one) | Value |
| --- | --- | --- |
|  | Introduction to <topic> | 5 |
|  | An overview of <topic> | 10 |
|  | Hands on/ How to of <topic> | 15 |
|  | Intermediate training for <topic> | 15 |
|  | Certification for <topic> | 20 |
|  | Advanced or Mastery of <topic> | 25 |

- Overview | Determining if Testing is Needed | **A Decision Tool** | Case Scenarios | Review & Resources

## Assessing Complexity – Step 2

| √ | Content Volatility: How often is your content likely to change or need update? (select one) | Value |
| --- | --- | --- |
|  | Every 3 months or less (Often) | 15 |
|  | Every 6 months (Moderate) | 10 |
|  | Every 12 months (Seldom) | 5 |

| √ | Audience: Your primary audience is estimated to be: (select one) | Value |
| --- | --- | --- |
|  | Researchers (Medical / Clinical / Lab technicians) | 5 |
|  | IT (Programmers, Database Architects / Modelers, Informaticians) | 5 |
|  | Support Staff (Administrators, Study Coordinators, Project Managers) | 5 |
|  | Mixed – could include any of the above | 15 |

- Overview | Determining if Testing is Needed | **A Decision Tool** | Case Scenarios | Review & Resources

## Assessing Complexity – Step 3

- If sum total Complexity > 90 = ‘High’
- between 45-90 = ‘Med’
- < 45 = ‘Low

| √ | Course Length (Time):  The total amount of time estimated to complete the course  (select one) | Value |
| --- | --- | --- |
|  | 1 hour or less | 5 |
|  | Between 1-2 hours | 10 |
|  | Between 2-4 hours | 15 |
|  | 4 hours or greater | 20 |

| √ | Training Type:  Your training type is: (select one) | Value |
| --- | --- | --- |
|  | Classroom | 5 |
|  | Virtual Classroom | 5 |
|  | Mentoring | 5 |
|  | Blended | 10 |
|  | Self-Directed | 15 |

- Overview | Determining if Testing is Needed | **A Decision Tool** | Case Scenarios | Review & Resources

## Assessing Feasibility

| √ | Resources available (costs) – Check all that apply | Value |
| --- | --- | --- |
|  | You or your institution has a SCORM authoring tool available to use | 10 |
|  | You have 2 or more staff available to develop material | 10 |
|  | You have only one staff person available to develop material | 5 |
|  | You do not have a SCORM authoring tool | 5 |
|  |  |  |

- If sum  = 20 then  ‘High’
- 15  = ‘Med’
- 10  = ‘Low

- Overview | Determining if Testing is Needed | **A Decision Tool** | Case Scenarios | Review & Resources

## Risk + Complexity + Feasibility

| Total Score | Testing Recommendation |
| --- | --- |
| 35 – 75 | Consider 
Self-tests, Take-home, or In-class exercises, or no tests |
| 80 – 155 | Consider
Self-tests, Take-home, or In-class exercises or Qualification level tests |
| 160– 195 | Recommend
Qualification level testing |
| 200 ↑
 | Require
Qualification level testing |

- Sum all 3 scores together

- Overview | Determining if Testing is Needed | **A Decision Tool** | Case Scenarios | Review & Resources

## Case Scenarios

- Example: *caCORE 1060: “Creating Metadata from Forms”*
- 3 Parts (1hr 45 min each)
- Instructor led virtual classroom
- Objectives (from LMS system):
- Describe how Data Elements are formed
- Decompose questions on a form into metadata
- Use of the CDE Curation Tool and the Admin Tool to create Data Elements derived from Forms, including EVS term submissions.
- Assessment Results click here 

- Overview | Determining if Testing is Needed | A Decision Tool | **Case Scenarios** | Review & Resources

## Case Scenarios - caCORE 1060

- Risk = 115(High), Complexity = 60(Med) , Feasibility = 20(High)
- Total=**195**

| Total Score | Testing Recommendation |
| --- | --- |
| 35 – 75 | Consider 
Self-tests, Take-home, or In-class exercises, or no tests |
| 80 – 155 | Consider
Self-tests, Take-home, or In-class exercises or Qualification level tests |
| 160– 195 | Recommend
Qualification level testing |
| 200 ↑
 | Require
Qualification level testing |

- Overview | Determining if Testing is Needed | A Decision Tool | **Case Scenarios** | Review & Resources

## Case Scenarios

- Example: *Pathway Tools: “Pathway Tools Hands-On Training* *”*
- Length (approx 1 hr)
- Self-paced, computer based training
- Objectives (from LMS system):
- Describe the components of Pathway Tools so the user gets a feel for the system
- Go through the installation of  the database component, cPath, and import data into the database
- Use Cytoscape and cPath to visualize pathways
- Overlay gene expression data and visualize the results using Cytoscape
- Assessment Results click here 

- Overview | Determining if Testing is Needed | A Decision Tool | **Case Scenarios** | Review & Resources

## Case Scenarios - Pathway Tools 

- Risk = 5(Low), Complexity = 50(Med), Feasibility = 10(Low)
- Total=**65**

| Total Score | Testing Recommendation |
| --- | --- |
| 35 – 75 | Consider 
Self-tests, Take-home, or In-class exercises, or no tests |
| 80 – 155 | Consider
Self-tests, Take-home, or In-class exercises or Qualification level tests |
| 160– 195 | Recommend
Qualification level testing |
| 200 ↑
 | Require
Qualification level testing |

- Overview | Determining if Testing is Needed | A Decision Tool | **Case Scenarios** | Review & Resources

## Review -  A Recap

- When deciding whether to test or not, consider the following criteria.
- Risk
- Potential for harm
- Complexity
- Course objectives
- Content Volatility
- Your Audience
- Type of Training
- Feasibility
- Resources available

- Overview | Determining if Testing is Needed | A Decision Tool | Case Scenarios | **Review & Resources**

## Review -  Questions

- My training content is high risk but my resources are very limited – what are my options?
- Deliver and administer via paper (though this may prove to be more expensive in the long run).
- Mitigate the risks (i.e. following training, allow access to test systems instead of production)
- NCICB’s Learning Management System does have capability to build a computer based tests.  For more info, contact: _[ncicb@pop.nci.nih.gov](mailto:ncicb@pop.nci.nih.gov)_

- Overview | Determining if Testing is Needed | A Decision Tool | Case Scenarios | **Review & Resources**

**Notes:**
Jenny – what link or e-mail should I place here?

## Review -  Questions

- Our total score placed our training in the *“*_*Consider Self-tests, Take-home or In-class exercises, or Qualification level tests*_*”* category – how do we decide which of these choose?
- This decision is largely a matter of preference for the training developers but the elements of Complexity and Feasibility should factor in, for example:
- If the course material is lengthy (4+ hours) but you plan on classroom delivery, then in-class exercises would be a suitable choice as they would provide variety and break between lectures.
- However, if you are planning to deliver in an hour or less, then take-home exercises may be a better choice since time is limited.
- If you have chosen a Self-Directed delivery method (i.e. CBT) and you have high feasibility (many resources) then creating a Qualification level type test may be a good choice (even if it’s low risk) as a means to keep students engaged and interested when there is no direct interaction with instructor or other students .

- Overview | Determining if Testing is Needed | A Decision Tool | Case Scenarios | **Review & Resources**

##  References & Resources

- _Developing caBIGTM Training: Training Approaches  & Testing_.  D & T Workspace.
- Westgaard, Odin.  _Designing and Delivering Fair and Practical Measurement Tools in the Workplace_.  San Francisco, CA.  Jossey-Bass/Pfeiffer; 1999
- Documentation & Training (D & T) Workspace Members (12/06/2006)
- American Society for Training and Development.  _[http://www.astd.org](http://www.astd.org/)_ (accessed 10/18/2006)
- Hale, Judith.  _Performance-Based Certification: How to Design a Valid, Defensible, and Cost Effective Program_.  San Francisco, CA. Jossey-Bass/Pfeiffer; 2000
- Free Management Library: Overview of Training and Development. _[http://www.managementhelp.org/trng_dev/trng_dev.htm](http://www.managementhelp.org/trng_dev/trng_dev.htm)_ (accessed 10/18/2006)

- Overview | Determining if Testing is Needed | A Decision Tool | Case Scenarios | **Review & Resources**