Webinar Series

Accredited Credential Programs- Building Trust Between Employers

and Credential Providers Through Rigorous Assessments

May 19, 2021
cete.osu.edu

CETE

Accredited Credential Programs: Building Trust Between Employers and 

Credential Providers Through Rigorous Assessments

NICE Webinar, 2021 (May 19)

James T. Austin & Bridget McHugh

Center on Education and Training for Employment

1
cete.osu.edu

CETE

Overview

◉ As IT credentialing has increased over 30 years , what are some trends?

• Credential Engine an emerging repository (ANSI-ANAB, Lumina, & collaborators)

• Accreditation (voluntary) strengthens certification programs

• Certification used by vendors, organizations, and government (DoD, IC, etc.)

◉ Grounds for concern

○ Confusion over standardization & types of credentials (certificates vs certifications)

○ Issues around credential meaning, content, quality, validity, comparability, and 

relationships between credentials covering same space

◉ Initiatives / innovations to address concerns

• NICE framework – revised with addition of competencies – is MAJOR 

• Accreditation (voluntary) as a key avenue for evaluating credential quality

◉ Solar Winds / Colonial Pipeline show why skilled cybersecurity workforce is vital**

2
cete.osu.edu

CETE

Our Learning Goals:

◉ Understand credentials and accreditation (quality)

◉ Which credential (certificate / certification) is appropriate for CTE / 

K12 cybersecurity pathways leading to PS education / Careers

◉ Overview creation of valid / reliable assessments used for 

certification (NOTE: more is needed for accreditation)

◉ Identify elements of ISO 17024 (2012) and NCCA (2016) standard 

and the processes managed by accreditation bodies

◉ Discuss stakeholder collaborations on CTE / K12 emerging 

workforce

3
CETE

cete.osu.edu

“

Credentialing is the process to award some 

verifiable indicator that a student or person has 

demonstrated qualification / competency in a 

content area (IT Career Field, Cybersecurity 

Pathway)

What is credentialing?

4
CETE

cete.osu.edu

Types of Credentials

◉ Credentials take a variety of forms. They include: 

○ academic degrees at multiple levels (CS; familiar to faculty)

○ state licenses (mandatory for practice)

○ training certificates and assessment-based certificates (ABC)

○ continuing education units (CEU)

○ board certifications (prominent in health sciences)

○ internal (vendor-, factory-) credentials (prominent in IT/ Cyber)

○ digital badges (discrete learning, & certification / recertification) 

◉ Our focus is certifications in Cybersecurity

5
CETE

cete.osu.edu

Why Use Credentials?

◉ Credentialing can “signal” info relevant to work performance:

• Knowledge & Skills, Competencies (NICE Framework)

◉ Provides data during transition points (HS – PS – Labor Market)

◉ Adelman’s (2000) “parallel PS universe” emphasized IT 

• HS to PS to labor market (career-spanning)

◉ Enables lifelong learning, supports “stacking” of credentials

◉ Remember: credentials are NOT “created equal” and there can be 

significant differences in quality, meaning, and perception

6
CETE

cete.osu.edu

Certification vs. Certificate Programs

Certifications Certificate Programs

Focus • Verifying education/ training & experience 

obtained elsewhere

• Assessing current KSA

• Maintenance: learning / competence

• Educating/training individuals to achieve 

intended learning objectives (ILOs)

• Assessing attainment of ILOs 

Core 

Elements

• Eligibility requirements (prerequisites)

• Assessment + maintenance of competence

• Learning event(s) MAJOR FOCUS

• ILO attainment assessed

Attributes • Can be revoked

• Designation & associated acronym 

• Cannot be revoked

• No designation/ acronym

Content 

Domain

• Based on formal job/practice analysis 

(occupation across settings) with 

verification 

• Based on curriculum, derived from 

procedures that may be formal

Cyber-Ed 

Strategies

• Orient to desirable creds & prereqs

• Determine preparation strategies

• Design certificates around NICE & 

consider accreditation standards

Certificate vs Certification

7
CETE

cete.osu.edu

Credentialing Examinations

◉ Chain of content evidence from job task analysis to blueprint to item development to 

maintenance of the item bank (supporting > 1 “forms” of exam)

○ Scores are criterion-referenced to NOT norm-referenced

◉ By examination score, “passing” persons showed their knowledge / skill by meeting a 

practitioner-recommended cutoff (typical range: 65-80%), while “failing” did not

○ Passing & failing levels ordered in their knowledge-skill (“warrants”)

◉ Certification criteria: apply, meet prereqs., take / pass exam(s), accept ethics code

◉ Purposes: valid/ reliable assessment of “Required Competence” (blueprint)

○ Industry knowledge (IT Basics, Cyber – Penetration Tests, Incident Handling); 

○ Key occupation-related skills, abilities, and relevant competencies

◉ Format: Knowledge (multiple choice), practical (performance), combination (multiple 

hurdle or composite)

8
CETE

cete.osu.edu

Credentialing Examinations

◉ Exams are purpose-built to evaluate occupational competency (fairly, validly)

○ Initial framework / scheme requires scope/ purpose and content domain 

○ Nutshell: Job-Task Analysis consists of a panel, followed by a verification 

survey (“survey says”) and panel review of results

■ Panel develops task statements (grouped) to reflect WHAT is done

■ Panel develops knowledge / skill (KS) to reflect HOW tasks done

■ NICE framework with components rated (via survey) would qualify

■ (ISC)2 defines and revises Common Body of Knowledge using JTA

■ Surveys: background info, task & K-S ratings, global (Qualtrics or other)

■ Response rates, missing data, & repeated statements used to evaluate

○ Analyses yield weights (# items for duties/ tasks or K-S), often crossed with 

cognitive rigor (recall, application, analysis / evaluation is a common system)

9
CETE

cete.osu.edu

Credentialing Examinations

◉ Blueprint represents the transfer of work domain to exam development, guides item 

writing by establishing composition of item (question) bank to support exam forms

◉ Aids communication with stakeholders (employers, educators, learners) – how?

○ Employers could: compare their job analyses (NOT job descriptions) to the 

body of knowledge / skill in the certification, evaluate value proposition 

○ (Local, State, & National) educators could: ensure that scope & sequence of 

educational-technical content standards align to certification BoK

○ Learners could: review candidate handbooks at certifier website to see if they 

qualify, how to prepare

◉ What could certifying bodies do to support these three stakeholder populations? Let 

us count the ways!!

10
CETE

cete.osu.edu

Credentialing Examinations

◉ Item Writing (IW) remote or in-person (pandemic balanced), test security crucial

○ Knowledge-cognitive skill measured by MC questions 

■ Rigor can be enhanced using scenarios, figures, and other “auxiliaries”

○ Practical “items” created from task(s) with skill as the psychological construct 

but REQUIRE a scoring guide for assessors to evaluate performance

○ Combos must balance coverage of blueprint/content outline (duplication)

◉ Item review occurs after item development: 

○ Edits / graphics, Subject Matter Expert ratings, psychometric analysis of ratings

◉ ALWAYS write more items than needed to support multiple forms (equated):

○ Ratings and field test data yield stats for form selection (match blueprints)

○ Keep items in a secure database with programming (queries, meta-data tagged 

to item – difficulty (p), item differentiation, “exposure”, “option” functioning, etc.

11
CETE

cete.osu.edu

Credentialing Examinations

◉ Passing point (cutoff, cut score) is very important because it is the “standard” against 

which exam performance is judged – judgments informed by data

○ Focus usually on minimal / borderline competence (rather than top-down)

○ Balance correct decisions & decision errors (False Positive & False Negative)

◉ Many, many methods exist (Cizek, 2012 provides a recent review)

○ Angoff variations are probably most common, defensible

○ Bookmark methods are increasingly used

○ Extended Angoff, analytic judgment, body of work, & dominant profile

◉ Recruit a panel (N>=10), train panelists on minimal competence and allow for 

practice / sharing, make TWO rounds of ratings with discussion between (sometimes 

additional information is provided during Round 2, analyze judgments, submit 

recommendations to higher authority, create technical documentation 

12
CETE

cete.osu.edu

Credentialing Examinations

◉ Resources for credentialing examination development: 

several key sources

○ Lane, Raymond, & Haladyna (2015). Handbook of 

test development (2nd ed.)

○ Henderson (2019). Certification: The ICE handbook 

(3rd ed.) – single best resource (IMHO) 

○ Fein (2012). Test development.

◉ As stated earlier, certification body websites provide 

information for candidates, employers, educators, etc.

○ (ISC)2 is at https://www.isc2.org/

◉ Psychometric or delivery vendors: ACS Ventures, Alpine, 

Pearson VUE, Prometric, PSI, & Yardstick (YAS)

13
CETE

cete.osu.edu

SHORT BREAK #1

◉ Please submit any questions through the webinar functions

14
CETE

cete.osu.edu

Growing Trend: Accreditation

◉ Accreditation of Certification Programs (NOTE: ISO 17011 covers accreditation bodies)

○ Educators (higher education) familiar with program or institution accreditation (mandatory)

○ For certification progs, voluntary (accreditation bodies review applications, check for conformity)

○ Pros:

■ Standards and guidelines for program definition (road map): certifications or certificates

■ Quality assurance for certification or certificate credentials: if program chooses to apply

■ Federal level interested in accreditation for creds across government (DoD, IC)

○ Cons:

■ Alignment with K12 Cybersecurity Education-Training (key role for NICE & framework!!)

■ Difficult for HS students to meet prerequisites to sit for the examination (pre-HS diploma)

◉ Accreditation requires:

○ Policies & procedures, appropriate persons to conduct work, & documentation (vital)

○ Application process with documentation and auditing / review (see below for steps)

○ If accreditation granted, “surveillance” of certification bodies requires annual reports to accreditor

15
CETE

cete.osu.edu

Emerging Trends

Accrediting Body Certificate Programs Certification Programs

ANSI (since 2003) 

https://anab.ansi.org

/credentialing/

ASTM E2659-18

Std Practice for Certificate 

Progs

ISO/IEC 17024:2012 (updated from 2003 original)

Conformity assessment – General requirements for 

bodies operating certification of persons – 10 

clauses

NCCA (ICE)

(since 1977)

www.credentialingex

cellence.org/NCCA

ANSI/NOCA 1100 – (2019) 

Std for Assessment-Based 

Certificate Progs

NCCA Standards for the Accreditation of 

Certification Programs (2016) – 24 standards from 

Purpose to Renewal (currently under revision)

ICAC (since 1996)

www.icacnet.org/affo

rdable-accreditation/

N / A at this time ISO/IEC 17024:2012 

Conformity assessment – General requirements for 

bodies operating certification of persons – 10 

clauses

Key Accreditors for Certifications and Certificates

16
cete.osu.edu

CETE

ISO 17024 Accreditation

What are the key elements: Depends on “role”

◉ 10 clauses: certification scheme & process

○ Scheme = framework fit around “certification 

process” from application to suspension

○ Process = apply, test, decision to award

◉ Management system an important requirement

◉ Reference: Judd’s 17024 Compliance Handbook 

templates (served as an ANAB auditor)

17
cete.osu.edu

CETE

ISO 17024 Accreditation

1. Scope

2. Normative references

3. Terms and definitions

4. General requirements (important, but outside our purview)

5. Structural requirements (important, but outside our purview)

6. Resource requirements (important, but outside our purview)

7. Records and information requirements (important, but outside our purview)

8. Certification schemes (focus today)

9. Certification process requirements (focus today)

10. Management systems requirements (important, but outside our purview)

Annex A - Principles

18
cete.osu.edu

CETE

ISO 17024 Accreditation

Scheme is the framework; Process is the steps for candidates

◉ Scheme includes 6 sub-clauses

○ 8.1 shall be a scheme (shall=must for ISO)

○ 8.2 elements: Scope to Code of Ethics

○ 8.3 process reqs: Initial Certification to Scope Change

○ 8.4 documentation required (membership, discussions, 

deliberations, votes & decisions of SC)

○ 8.5 reviewed-validated on an on-going, systematic basis

○ 8.6 scheme ownership requirements

19
cete.osu.edu

CETE

ISO 17024 Accreditation

Scheme is the framework; Process is the steps for candidates

◉ Elements of a scheme / framework (8.2):

○ Scope of certification (Name/designation, level, population, etc.)

○ Job-Task Description: JTA with verification (very important)

○ Required competence (Exam Blueprint+ Knowledge-Skill)

○ Abilities (Physical, Psychomotor, Cognitive)

○ Prerequisites (education, experience, etc.)

○ Code of Conduct / Ethics

20
cete.osu.edu

CETE

ISO 17024 Accreditation

Scheme is the framework; Process is the steps for candidates

◉ Process requirements of a scheme / framework (8.3)

○ Criteria for INITIAL certification & recertification

○ Assessment Mechanism for INITIAL certification & recertification 

(plus a blueprint)

○ If applicable, SURVEILLANCE methods / criteria

○ Criteria for SUSPENDING or WITHDRAWING certification

○ If applicable, criteria for CHANGING SCOPE/LEVEL

21
cete.osu.edu

CETE

ISO 17024 Accreditation

Scheme is the framework; Process is the steps for candidates

◉Other requirements of a scheme / framework (8.4 to 8.6)

-8.4 documentation required (membership, 

discussions, deliberations, votes & decisions of committee)

-8.5 reviewed-validated on an on-going, systematic 

basis

-8.6 scheme ownership requirements (types of 

owners)

22
cete.osu.edu

CETE

ISO 17024 Accreditation

Clause 9: Process covers steps for candidates

◉ 9.1 Application process 

◉ 9.2 Assessment process 

◉ 9.3 Examination process 

◉ 9.4 Decision on certification

◉ 9.5 Suspending, withdrawing, or reducing the scope of certification 

◉ 9.6 Recertification process

◉ 9.7 Use of certificates, logos and marks

◉ 9.8 Appeals against decisions on certification 

◉ 9.9 Complaints 

23
CETE

cete.osu.edu

https://anab.ansi.org/credentialing/personnel-certification/how-to-apply

How to earn 

accreditation:

Stages of 

accreditation 

application 

process

24
CETE

cete.osu.edu

https://anab.ansi.org/credentialing/personnel-certification/how-to-apply

25
cete.osu.edu

CETE

NCCA Accreditation

Institute for Credentialing Excellence (formerly NOCA, since 1977)

◉ Autonomous: National Commission on Certifying Agencies is accrediting body 

○ Commissioners represent administration/operations and psychometrics

◉ Published 2016 standards for accreditation (under revision, expected 2021-22) 

◉ Certifying bodies submit applications (3 windows / year), internal reviews conducted 

(adequate or inquiry), then applications are reviewed / voted upon by Commissioners 

○ If conformity established, accreditation is awarded

○ If non-conformity exists for one or more standards, time-limited process to correct

○ Accreditation, like certification, granted for a specific “term” (5 years is common for 

accreditation, while 3-5 years is common for certifications)

◉ Conducted Remote Live Proctoring pilot, also allowed exemptions during 2020-21

26
cete.osu.edu

CETE

NCCA Accreditation

Institute for Credentialing Excellence (formerly NOCA, since 1977)

◉ 24 NCCA Standards (2016) are grouped into 5 categories

○ (1-5) Purpose, Governance, & Resources

○ (6-12) Responsibilities to Stakeholders

○ (13-21) Examinations (SME Panels, JTA, Test Specifications, 

Item Writing, Standard Setting, Test Administration, ScoringReporting, Reliability, Examination Equating)

○ (22-23) Maintaining Certification

○ (24) Maintaining Accreditation

27
cete.osu.edu

CETE

NCCA Accreditation

◉ NCCA standards comparable to the ISO 

standard, but provide specificity (essential 

elements & commentary) 

◉ ICE is an organization bringing together 

stakeholders (certification bodies, 

vendors, etc.); ANAB is accreditor

◉ ICE offers Credentialing Specialist training 

(certificate) and is creating a Credentialing 

Professional Certification

28
cete.osu.edu

CETE

Examples: Accredited Cyber Certs

29
cete.osu.edu

CETE

Judgments Of Accredited Cyber Certs

In the certification space of Cyber, few are widely accepted as "Industry Standard"

1.Certified Ethical Hacker (CEH)

2.CompTIA Security+

3.Certified Information System Security Professional (CISSP)**

4.Certified Information Security Manager (CISM)

5.Certified Information Systems Auditor (CISA)

6.NIST Cybersecurity Framework (NCSF)

7.Certified Cloud Security Professional (CCSP)

8.Computer Hacking Forensic Investigator (CHFI)

9.Cisco Certified Network Associate (CCNA)

30
CETE

cete.osu.edu

SHORT BREAK #2

◉ Please submit any questions through the webinar functions

31
CETE

cete.osu.edu

People Behind Scenes

◉ Best exams driven by certification bodies who manage teams of experienced 

practitioners, psychometricians, IT, and other specialists

◉ My prototype reflective / experienced practitioner: Tom O’Neill, Butler Career & 

Technology Schools (Butler Tech) in SW Ohio (networks, cyber among his courses)

○ Tom an instructor since 2001, provides PS level information in HS classes at 

Ross High School (satellite location of Butler Tech)

○ Item writer for state testing system (we met “back in the day”)

○ Struck by rigor (Tom’s items) and reactions (peer instructors)

○ Provides real world experience to his students (PS+RW)

○ Coaches award-winning student teams in competitions

◉ NOTE: Tom was warned in early May about his “prototypicality”!!

32
CETE

cete.osu.edu

People Behind Scenes

◉ Best exams driven by certification bodies – teams 

of seasoned-novice practitioners, IT, 

psychometricians, and many other specialists

◉ Prototype reflective / experienced psychometrician: 

Liberty Munson, PhD

◉ Liberty’s doctorate from University of Illinois (I-O 

Psychology, Psychometrics)

○ Began with Microsoft after graduate school

○ Current position: Director of MS Certification 

programs (new areas: Azure, Power Platform)

○ Contributor to field (ATP, ICE, WorkCred, 

...........), mentor,

Who is this psychometrician?

33
CETE

cete.osu.edu

People Behind Scenes

◉ Accreditation bodies work in teams as well 

◉ Prototype reflective / experienced accreditation 

director: Linda Anguish is the director for I.C.E. 

◉ Linda’s background featured work for a certification 

body (accredited, naturally) 

◉ In her current role, Linda works with other I.C.E. 

staff and consultants, the NCCA Commissioners, 

and certification bodies before, during, and after 

their submission – she is a hub for many

◉ Involved in revising the NCCA Standards (ongoing)

Accreditor

34
CETE

cete.osu.edu

Stakeholders in Credentialing

Credentials are widespread, 

stakeholders are numerous, and 

value propositions may vary

We focus on relationships between:

--Employers

--Education (CTE, K12)

--Certification bodies

35
CETE

cete.osu.edu

Educational

Bodies

Trade 

Associations Certification 

Bodies

National 

Standards 

Organizations

Employers/

Employees

36
CETE

cete.osu.edu

Emerging Interest in Credentialing Stakeholders in Credentialing

Employers- Employers often look at degrees / certifications as indicators of competence, and may use them in hiring, 

promotion, & placement decisions. Certification includes healthcare, personal fitness, & information technology

Education Community- produces potential certification holders, Career-Technical & K-12 programs with pathways to 

PS education and Cyber careers

Certification Bodies- Major hardware / software manufacturers (MS, Cisco), certification bodies such as (ISC)2; 

CompTIA (vendor-neutral) 

Trade Associations- Organizations representing membership, including Institute for Credentialing Excellence 

(I.C.E.) and Association of Test Publishers (ATP; Divisions: Certification, Licensure & Workforce Credentialing, etc.)

National Standards Organizations- American National Standards Institute (ANSI) is the U.S. member of ISO and 

has authority to designate Standards Developing Organizations (SDO); ANSI National Accreditation Board (ANAB) 

handles the accreditation activities for both certifications (focus) and certificate programs (very brief coverage) 

37
CETE

cete.osu.edu

Stakeholder Collaborations Emerging Trends

◉ Credentialing interest (signaling independent of degrees, “parallel universe”)

○ Career start without post-secondary degree (through self-study, training) 

– parallel universe

○ How can K12 (HS) education enter the dining hall (be “at the table”)?

◉ How can stakeholders work together better?

○ Value for each stakeholder group collaborating?

■ Certification bodies increase awareness in HS populations (market)

■ PS as a “bridge” between HS and career, not necessarily

■ Certification bodies do not want to compromise exams, but could 

evaluate prerequisites (education, experience) to minimize barriers

38
CETE

cete.osu.edu

Stakeholder Collaborations Emerging Trends

◉ Government is another key stakeholder

○ NICE is a prime example

◉ Bipartisan bills introduced in House and Senate to allow 529 College Savings 

Plan funds to pay certification and recertification costs

○ Freedom to Invest in Tomorrow’s Workforce Act

◉ State governments are approaching credentials by paying for costs if the 

candidate earns the certification

39
CETE

cete.osu.edu

Emerging Trends Conclusions

◉ Credentialing(independent of degrees), focus is on persons

○ Career start without PS degree (self-study, training) – parallel universe

◉ No vetted value proposition for certification or accreditation for EMPLOYERS

○ Do cred OR accredited cred help bottom line? Predict job performance? 

○ Value of certification & salary surveys suggest positive relationship

○ Accredited credentials are assumed better by most stakeholders

◉ Accreditation expense (expensive although this varies by accreditor)

◉ NCCA ~$2655; does not use auditors (can prep ISO 17024 WorkCred, with 

NICE input, built a framework for certs within Bachelors-level degrees, but 

articulating to HS remains problematic

◉ For HS: Screen certs for accreditation prior to recommending

40
CETE

cete.osu.edu

Questions?

DISCUSSION

41
CETE

cete.osu.edu

Contact Information

James Austin, PhD 

614-292-9897

Austin.38@osu.edu

Assessment Program Lead

Bridget McHugh, PhD 

614-514-5865 

McHugh.159@osu.edu

Psychometric Team Lead

42
Thank You for Joining Us!

Upcoming Webinar:

“Digital Citizenship: Safety and Security for an Online World”

When: Wednesday, June 16, 2021, 2:00-3:00PM ET

Register:

https://nist-nice.adobeconnect.com/webinarjune2021/event/registration.html

nist.gov/nice/webinars

43