Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 1 of 11

Purpose

The purpose of this document is to provide supplemental guidance on writing effective 

multiple choice questions to examiners and licensees who develop and review Operator 

Licensing written exams. 

Scope

This document provides clarifications and guidance for fulfilling the intent of 10 CFR 55 

Subpart E and NUREG 1021, as they pertain to written test items. The use of this document 

is not a regulatory requirement and shall be used on a voluntary basis. The following 

provisions are applicable: 

1. This document does not impose any requirements or expectations on licensees 

beyond those in NUREG 1021. 

2. This document does not replace or eliminate the requirements within NUREG 1021. 

3. Anyone discovering a conflict between this guidance and NUREG 1021 should 

consider bringing it to the attention of the responsible NRC Region Office or the 

Headquarters (HQ) Program Office. 

4. NUREG-1021 always takes precedence if a conflict is identified. 

References

1. K/A catalogs (NUREG 1122 and NUREG 1123) 

2. 10CFR55, Subpart E - Written Examinations and Operating Tests. 

3. NUREG 1021, Rev. 9, Supplement 1, “Operator Licensing Examination Standards For 

Power Reactors” Appendix B 

4. Operator Licensing Feedback Web page questions 401.2, 401.3, 401.16, 401.48, and 

Gen 8 @ http://www.nrc.gov/reactors/operator-licensing/op-licensing-files/olfeedback.pdf

5. MANTG/NRC Conference August 2005 & Licensee Operator Licensing Workshop 

presentation (ADAMS Accession Number ML052790507) 

6. 2007 MANTG – Flawed Test Items (ADAMS Accession Number ML073130392) 

7. 2008 Region III NRC & Licensee Operator Licensing Workshop presentation (ADAMS 

Accession Number ML090970297) 

8. 2011 Region II Exam Writers’ Conference, “the Subset Issue Correcting a Common 

Psychometric Flaw” (ADAMS Accession Number ML112240240) 

Table of Contents

• Section I: What is an Implausible Distractor? 

• Section II: Examples of Implausible Distractors 

• Section III: Making Distractors More Plausible 

• Section IV: Examples of Improved Multiple Choice Questions 

 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 2 of 11

I. What is an Implausible Distractor? 

Plausible: reasonable, appearing worthy of belief

Implausible: provoking disbelief

[Source: Merriam Webster’s Collegiate Dictionary - Tenth Edition] 

An implausible Distractor can be defined as an answer choice that can be quickly 

eliminated by someone with little or no knowledge of the topic being tested. For example, 

an answer choice could be considered implausible if it: 

 is composed of relatively unimportant information along with other choices 

that focus on more important concepts (trivial Distractor); or 

 does not grammatically follow from the stem (specific determiners); or 

 contains words such as “always” or “never” which suggest an incorrect 

option. 

NUREG-1021, Revision 9, Supplement 1, ES-401, Section C.2.b, states that an 

implausible Distractor is a “psychometric flaw.” 

Form ES-401-9, Written Examination Review Worksheet, Item 3 states that single 

implausible Distractors should be repaired; more than one is unacceptable. 

II. Examples of Implausible Distractors 

A. Requires Minimal Plant Knowledge

 

Which one of the following will cause the RHR pumps to start during a 

large break LOCA? 

A. low drywell pressure 

B. high reactor water level 

C. high drywell pressure

D. MSIVs in the NOT OPEN position 

Distractors A, B, and D are implausible because the choices can be all 

represent safe conditions when compared to Choice C; therefore, the 

question can be answered with minimum knowledge of the initiating 

signals for the LOCA logic. 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 3 of 11

B. Fails Common Sense Test

 

Why does the procedure titled "Reactor Refueling Operations" allow 

stopping all operating RHR pumps for up to 1 hour per 8-hour 

period during the performance of core alterations? 

A. To allow for swapping of RHR trains in service. 

B. To facilitate movement of fuel or core components. 

C. RHR cooling is not required at all by Technical Specifications with 

the pool flooded. 

D. To obtain Spent Fuel Pool heat up data to verify Time to Boiling 

calculations. 

Distractor C is not plausible because it is unrealistic for RHR cooling being 

not required at all by Technical Specifications while in refueling, especially 

when one considers boron mixing, decay heat removal, etc. 

Given the following conditions: 

• Reactor Power is 100% 

• Pressurizer level is 56% and rising 

• Pressurizer pressure is 2235 psig and slowly rising 

• Letdown flow isolates 

• Charging flow rises 

• Annunciator 32B, PZR 17% HTRS OFF, alarms 

• Annunciator 32C, PZR LO LEV DEV, alarms 

• All containment parameters are normal 

Which ONE (1) of the following has occurred? 

A. Reactor Coolant System Leak. 

B. Pressurizer Level Channel Failure. 

C. Loss of Charging. 

D. Letdown Line Break. 

Distractors A, C and D are not plausible because the 2nd bullet in the stem 

indicates that pressurizer level is slowly rising. 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 4 of 11

C. Physics not Correct

 

Given the following plant conditions: 

• Reactor power is 100% 

• Pressurizer pressure channel I(429), has been removed from service 

for surveillance testing with its’ associated bi-stables tripped 

• Pressurizer pressure channel IV (449), fails LOW 

Which ONE of the following describes the initial automatic system response? 

A. Reactor trip, but NO Safety Injection; 

Pzr PORV PR-2A remains closed 

B. Reactor trip and Safety Injection; 

Pzr PORV PR-2A remains closed 

C. Reactor trip but NO Safety Injection; 

Pzr PORV PR-2A opens 

D. Reactor trip and Safety Injection; 

Pzr PORV PR-2A opens 

The second portion of Distractors C and D is not plausible because the 3rd bullet 

that states that the Pressurizer pressure channel fails low; there is no reason for 

a Pzr PORV to open 

Torus water temperature rises from 75 °°F to 95 F over several weeks due to 

summer heat. 

Which of the following describes the effect of the rise in torus water 

temperature? 

A. The INCREASE in torus airspace would result in LOWER post-LOCA 

peak drywell pressure. 

B. The DECREASE in torus water level would result in LOWER available 

NPSH for the ECCS pumps. 

C. The DECREASE in torus airspace would result in HIGHER postLOCA peak drywell pressure. 

D. The INCREASE in torus water level would result in HIGHER available 

NPSH for the ECCS pumps. 

Distractors A and B are not plausible since it is commonly known that water 

expands when heated over this temperature range; therefore, an increase in 

torus air space or a decrease in torus water level is impossible. 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 5 of 11

D. Distractors Conflict with Information in Question Stem

E. Distractors Not Independent From Each Other (Subsets)

This is the most common plausible Distractor problem, but is the easiest to fix by 

using an appropriate modifier to eliminate the “subset”. 

 

The plant is operating at 100% power. 

A failure of the governor/pressure regulator occurs which causes the turbine 

control valves to fully open. 

Given the above conditions, which one of the following RPS trip signals will 

cause an automatic scram? 

A. Main Steam Isolation Valve Closure 

B. APRM flux - Upscale 

C. Low RPV water level 

D. Turbine Control Valve Closure 

Distractor D is not plausible because the stem states that the turbine control 

valves have failed open. 

Additionally, Distractor B is minimally plausible because an increase voids 

always adds negative reactivity. 

The Plant is in MODE 3. 

Which one of the following is the Reactor Coolant System (RCS) 

UNIDENTIFIED leak rate that results in LCO 3.4.13, RCS Operational 

LEAKAGE, to be NOT met? 

A. Any nonzero leak rate. 

B. 0.2 gpm. 

C. 1.1 gpm. 

D. 10.1 gpm. 

If one erroneously believed NO IDENTIFIED LEAKAGE is allowed, all 4 

answers would be correct because Distractors B, C, and D are subsets of 

Distractor A. To eliminate the subset issue, Distractor A should be revised to 

“Any nonzero leak rate < 0.2 gpm.” 

Similarly, since IDENTIFIED LEAKAGE must be limited to 1.0 gpm, Distractor 

C is correct. However, Distractor D is also correct because both answers are 

greater than 1 gpm. 

This can be fixed by adding the modifier “MINIMUM” to the stem. 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 6 of 11

The final version: 

III. Making Distractors More Plausible 

Distractors can be plausible by incorporating misconceptions about the topic. For 

example, in the following question pertaining to loss of subcooling margin, 

Distractors A, C, and D use these common misconceptions to create plausible 

Distractors. 

 

Which one of the following identifies the reason why the reactor 

coolant pumps (RCPs) are required to be secured during a small 

break LOCA when subcooling margin is lost? 

A. to prevent pump damage resulting from operation under 

two-phase conditions 

B. to prevent core damage resulting from rapid phase 

separation upon subsequent loss of RCS flow 

C. to reduce RCS pressure by removing the pressure head 

developed by the RCPs 

D. to remove heat being added to the RCS by the operating 

RCPs 

The Plant is in MODE 3. 

Which one of the following is the MINIMUM Reactor Coolant System (RCS) 

UNIDENTIFIED leak rate that results in LCO 3.4.13, RCS Operational 

LEAKAGE, to be NOT met? 

A. Any nonzero leak rate < 0.2 gpm. 

B. 0.2 gpm. 

C. 1.1 gpm. 

D. 10.1 gpm. 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 7 of 11

Another way to make Distractors more plausible is to use two part answer options. For 

example, in the following question, two part answer options of what and why are used. 

Distractor A is credible because for some LOCA events (such as the one described in 

the stem), core uncovery and superheated conditions are expected. 

Distractor C is credible because core uncovery and superheated conditions normally 

indicate inadequate core heat removal which requires transitioning to a Functional 

Recovery procedure. 

However, in this case, transitioning to a Functional Restoration is required because 

Safety Injection flow is only 535 gpm which is below the design flow of 600 gpm (150 

gpm per header) needed to adequately remove core heat. 

 

Given the following conditions: 

• A LOCA has occurred and the crew has entered OP-902-002, 

Loss of Coolant Accident Recovery 

• RCS pressure is 600 PSIA 

• HPSI pump A & B are running and all Safety Injection Flow 

Control valve are open 

• Cold leg injection header flow rates are as follows: 

o 1A is 190 gpm 

o 1B is 160 gpm 

o 1C is 175 gpm 

o 1D is 0 gpm 

• Fifteen minutes into the event, RVLMS indicates core 

uncovery and CETs are showing superheat. 

Based on the above indications, the crew should (1) because (2)? 

 (1) (2) 

A Remain in OP-902-002, 

LOCA Recovery 

Core uncovery and superheated 

conditions are expected 

B Remain in OP-902-002, 

LOCA Recovery 

The SI flow curves are met 

C Exit to OP-902-008, 

Functional Recovery 

Core uncovery and superheated 

conditions are NOT expected 

D Exit to OP-902-008, 

Functional Recovery 

The SI flow curves are NOT met 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 8 of 11

IV. Examples of Improved Multiple Choice Questions 

Westinghouse: System 061 Auxiliary/Emergency Feedwater 

K2.02 Knowledge of bus power supply to the following: AFW electric drive pumps

BEFORE (Minimal Plant Knowledge): 

AFTER:

This question is now an application level question because the applicant must predict the 

outcome of the event described in the stem. This question requires the applicant to recall 

automatic start signals to both the steam driven and the motor driven pumps in addition to 

power supplies to the motor driven AFW pumps. 

 

The plant is operating at 100% power with EDG #1 on clearance for a lube oil change-out. 

Subsequently, 

• An inadvertent reactor trip occurs COINCIDENT with a loss of offsite power. 

• All SG levels “Shrink” to 10% narrow range 

Which one of the following predicts the auxiliary feedwater pump that will be running including the 

electrical bus that will be providing the power to the pump? 

A. “A” AFW pump powered from 480V Bus 8N 

B. “A” AFW pump powered from 4KV Bus 2AE 

C. “B” AFW pump powered from 480V Bus 9P 

D. “B” AFW pump powered from 4KV Bus 2DF 

Distractors “A” and “B” can be easily eliminated by knowing EDG # 1 is the Train A 

emergency power supply. 

The plant is operating at 25% power with EDG #1 on clearance for a lube oil change-out. 

Subsequently, 

• An inadvertent reactor trip occurs COINCIDENT with a loss of offsite power. 

• All SG levels “Shrink” to 25% narrow range 

Assuming no operator actions are taken, which one of the following predicts the auxiliary feedwater 

(AFW) system response 

A. NO AFW pumps are running 

B. ONLY the Steam Driven AFW pump is running 

C. ONLY the “B” AFW pump is running 

D. BOTH the Steam Driven AFW pump AND “B” AFW pump are running 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 9 of 11

Westinghouse: System 063 DC Electrical Distribution 

K2.01 Knowledge of bus power supply to the following: Major DC loads 

BEFORE (Specific Determiners - Cueing): 

AFTER:

 

Which one of the following identifies the effect, if any, to DG-1A following a loss of DC bus11A? 

A. Loss of ALL engine protective tripping capability 

B. Loss of Normal and Emergency engine start circuits 

C. No effect, DG-1A remains FUNCTIONAL from bus 11C 

D. Loss of ALL engine operating parameters indications from any location.

Distractors A, C and D are weak Distractors due to the absolute nature of the 

responses (ALL, no effect, ALL) 

The plant is at 100% power and all systems are in a normal alignment. Power is lost to 

Vital 125 VDC bus 11A 

What effect, if any, will a loss of bus 11A have on DG-1A? 

A. The DG will NOT start on a Safety Injection signal 

B. The DG will start and run unloaded on a Loss of Offsite Power signal 

C. The DG will NOT start automatically or manually from the MCB, but can be started from the 

local diesel control panel. 

D. After DG-1A is started, it will NOT trip on an overspeed condition 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 10 of 11

GE BWR-4: Refueling Accidents

AK1.01 – Knowledge of the operational implications of the following concepts as they 

apply to REFUELING ACCIDENTS: Radiation Accidents 

BEFORE: Baseless Distractors – Distractors C & D use 26.5 feet. There is no reason 

behind this water level in the Accident Analysis, Technical Specification Bases or plant 

procedures. 

AFTER:

Distractors C & D use the Spent Fuel Pool Level (over the spent fuel racks) required by 

Technical Specifications. 

 

The plant is in MODE 5, REFUELING 

Which ONE of the following is the MINIMUM acceptable Water Level above the Reactor Vessel 

Flange, AND the reason for that limit? 

A. 20.5 feet provides adequate Iodine absorption following an accident. 

B. 20.5 feet provides adequate shielding of personnel during core alterations. 

C. 26.5 feet provides adequate Iodine absorption following an accident. 

D. 26.5 feet provides adequate shielding of personnel during core alterations. 

With the plant in MODE 5, REFUELING, with Core Alterations is progress. 

Which ONE of the following is the MINIMUM acceptable Water Level above the Reactor 

Vessel Flange, and the reason for that limit? 

A. 20.5 feet provides adequate Iodine absorption following an accident. 

B. 20.5 feet provides adequate shielding of personnel during core alterations. 

C. 22.0 feet provides adequate Iodine absorption following an accident. 

D. 22.0 feet provides adequate shielding of personnel during core alterations. 
Supplemental Guidance for Writing Effective Multiple Choice Questions 

Page 11 of 11

Westinghouse: System 015 DC Electrical Distribution 

AK3.07 Knowledge of reasons for the following response as they apply to the Reactor 

Coolant Pump Malfunctions (Loss of RC Flow): Ensuring that S/G levels are controlled 

properly for natural circulation enhancement. 

BEFORE: 

AFTER:

Distractors B and D are now more credible with containment temperature elevated. 

Otherwise, there is no reason to pick the control band associated with adverse 

containment parameters. 

Given the following: 

• Unit 3 was operating at power when a Loss of Offsite Power occurred. 

• The operating crew has implemented 3-EOP-ES-01, Reactor Trip Response. 

• Stable plant conditions have been completed. 

• Offsite power is expected to be restored within 24 hours. 

In accordance with 3-EOP-ES-0.1, which one of the following describes (1) the control band used for 

S/G levels and (2) the reason for that band? 

A. BETWEEN 15% AND 50% to preclude AFW re-initation and establish a heat sink 

which will enhance natural circulation. 

B. BETWEEN 32% AND 50% to preclude AFW re-initiation and establish a heat sink which 

will enhance natural circulation. 

C. BETWEEN 15% AND 50% to maintain an adequate inventory to start a natural 

circulation cooldown. 

D. BETWEEN 32% AND 50% to maintain an adequate inventory to start a natural 

circulation cooldown. 

Given the following: 

• Unit 3 was operating at power when a Loss of Offsite Power occurred. 

• The operating crew has implemented 3-EOP-ES-01, Reactor Trip Response. 

• RCS Pressure is 2210 psig and stable. 

• Pressuruzer Level is 35% and lowering. 

• Containment temperature is 105°F. 

• Offsite power is expected to be restored within 24 hours. 

In accordance with 3-EOP-ES-0.1, which one of the following describes (1) the control band used for 

S/G levels and (2) the reason for that band? 

A. BETWEEN 15% AND 50% to preclude AFW re-initation and establish a heat sink 

which will enhance natural circulation. 

B. BETWEEN 32% AND 50%% to preclude AFW re-initiation and establish a heat sink 

which will enhance natural circulation. 

C. BETWEEN 15% AND 50% to maintain an adequate inventory to start a natural 

circulation cooldown. 

D. BETWEEN 32% AND 50% to maintain an adequate inventory to start a natural 

circulation cooldown.