ATTACHMENT 65001.23

INSPECTION OF HUMAN FACTORS ENGINEERING VERIFICATION AND VALIDATION ITAAC

PROGRAM APPLICABILITY:  2503

65001.23-01    INSPECTION OBJECTIVES

01.01	To confirm by inspection that the combined license (COL) holder (licensee) has implemented Human Factors Engineering (HFE) verification and validation (V&V) activities in accordance with NRC approved implementation plans.  The inspection results will be used to support an NRC finding as to whether the V&V activities meet the acceptance criteria as stated in the AP1000 HFE Inspections, Tests, Analyses and Acceptance Criteria (ITAAC) Design Commitments 3.2.01a, b, c and d.

65001.23-02    INSPECTION REQUIREMENTS AND GUIDANCE

02.01	Background. 

Inspection of ITAAC associated with a COL is intended to support the Commission finding stipulated in 10 CFR Part 52.103(g), specifically that the ITAAC acceptance criteria have been met, and that the facility has been designed and built to conform to the licensing basis.  The Commission policy for Design Acceptance Criteria (DAC), as defined in SECY-92-053, “Use of Design Acceptance Criteria during 10 CFR Part 52 Design Certification Reviews,” allows licensees the option to provide HFE design process milestones as ITAAC.  In these instances the HFE DAC-related ITAAC are inspected as the HFE design activity is completed.

An HFE program, such as that described in NUREG-0711 (Reference 3), provides the structure for ensuring that the HFE aspects of a plant are developed, designed, and evaluated on the basis of a structured, disciplined analysis using accepted HFE principles.  V&V evaluations are that part of an HFE program which comprehensively determine whether the design conforms to HFE design principles and if it enables plant personnel to successfully perform their tasks to achieve plant safety and other operational goals. 

The integrated system validation (ISV) is the most important element of the V&V process and should receive the majority of inspection time. ISV is that part of the V&V process which evaluates an integrated system design (i.e., hardware, software, and personnel elements) using performance-based tests to determine whether the integrated system design acceptably supports safe operation of the plant. It is intended to evaluate the acceptability of those aspects of the design that cannot be determined through analytical means such as human-system interface (HSI) task-support verification and HFE design verification.  The ISV process is comprised of several important activities, including but not limited to, establishing an appropriate test facility, specifying test objectives, developing test scenarios, specifying performance measures, developing and implementing a test design (including pilot testing), and analyzing and interpreting the data.  

Issue Date:  12/22/14	1	65001.23

The ISV provides an opportune time to sample elements of the verification and corrective action activities.  Appendix 1 integrates the inspection of verification and corrective action activities into the ISV inspection to provide for an efficient ITAAC inspection.  

02.02   Inspection Requirements and Guidance

For each NUREG-0711 element applicants/licensees typically submit information in two parts: an implementation plan describing the process/method to be followed and a results summary report describing the design product.  The objective of this inspection is to verify that the licensee has completed V&V activities in accordance with the commitments contained in the approved implementation plan.

The actual planning and scheduling of this ITAAC inspection is dependent on:

a. Completion of the ISV inspection outlined in Appendix 1.  This inspection addresses completion of the HFE design and is typically conducted by Headquarters Vendor Inspection Branches because the work is conducted by a vendor.  

b. Receipt of the Results Summary Reports

02.03  Specific Inspection Criteria

a. Verify the inspection outlined in Appendix 1, Part 1 is complete and any issues identified have been resolved.

b. Complete the inspection outline in Appendix 1, Part 2

Guidance:  The team completing Appendix 1, Part 1 may have provided input on Appendix 1, Part 2 if sufficient testing data was available.  To complete Appendix 1, Part 2, additional inspection can be performed to supplement the results of the Part 1 inspection or a determination can be made that the information already obtained is sufficient. 

c. Verify the Results Summary Report content conforms to the ISV implementation plan. (See attachment 1 for ML number for this document.)

d. Using the information provided in the previous three criteria, verify the following objectives were met:

1. The role of plant personnel, i.e., that the allocation of functions to human and automatic aspects of the integrated system are appropriate and takes advantage of human strengths and avoid allocating functions that would be negatively affected by human limitations.

2. Shift staffing, assignment of tasks to crew members, and crew coordination (both within the control room as well as between the control room and local control stations and support centers) are acceptable.  This should include validation of the nominal shift levels, minimal shift levels, and shift turnover.

3. For each human function, the design provides adequate alerting, information, control, and feedback capability for human functions to be performed under normal plant evolutions, transients, design basis accidents, and selected risk significant events that are beyond design basis.

4. Specific personnel tasks can be accomplished within the time and performance criteria, with a high degree of operating crew situational awareness, and within acceptable workload levels that provide a balance between vigilance and operator burden. 

5. The operator interfaces minimize operator error and provide for error detection and recovery capability when errors occur.

6. The functional requirements are met for the major HSI features, e.g., group- view display, alarm system, safety parameter display system (SPDS) function, general display system, procedures, controls, communication systems, and emergency operating procedure (EOP)-related local control stations.

7. The crew can make effective transitions between the HSI features in the accomplishments of their tasks and that interface management tasks such as display configuration and navigation are not a distraction or undue burden.

8. The integrated system performance is tolerant of failures of individual HSI features.

9. Aspects of the integrated system (including staffing, communications, and training) that may negatively impact integrated system performance are identified.

02.04  Requirements for Performance of Inspection. 

The inspection will be performed in accordance with the inspection plan.  Adjustments to the inspection plan will be communicated to Region II Construction Inspection Staff and Operations Branches 1 and 2 to minimize impact to the licensee and to assist in revising inspection planning efforts accordingly.

02.05  Requirements for Inspection Reporting. 

An inspection report and any findings will be prepared and approved in accordance with Inspection Manual Chapter 0617.

65001.23-03    RESOURCE ESTIMATE

The total estimated hours to complete this inspection for one COL licensee is 300 staff hours for completion of the Appendix 1 inspection and 80 hours for the ITAAC closure inspection.

65001.23-04    REFERENCES

1.	10 CFR Part 52, “Licenses, Certifications, and Approvals for Nuclear Power Plants”

2.	Regulatory Guide 1.206, C.II.1.2.9, “Human Factors Engineering” and C.III.5, “Design

Acceptance Criteria”

3.	NUREG-0711, Rev 2, “Human Factors Engineering Program Review Model,” February,

2004

4.	NUREG-0800, Section 14.3.9, “Human Factors Engineering - Inspections, Tests, Analyses, and Acceptance Criteria”

5.	NUREG-0800, Section 18, “Human Factors Engineering”

6.	NUREG-1793, Supplement 2, “Final Safety Evaluation Report Related to Certification of the AP1000 Standard Design,” 2011 

7.	NUREG/CR-6393, “Integrated System Validation: Methodology and Review Criteria,”

(O'Hara et al., 1997)

8.	Inspection Manual Chapter 0613, “Documenting 10 CFR Part 52 Construction and Test

Inspections” 

9.	ANSI/ANS 3.5 – 1998, “Nuclear Power Plant Simulators for Use in Operator Training and Examination,” (American National Standards Institute, 1998)

10.  ANSI/ANS 3.5 – 2009, “Nuclear Power Plant Simulators for Use in Operator Training and Examination,” (American National Standards Institute, 2009)

11.  SECY-92-053, “Use of Design Acceptance Criteria during 10 CFR Part 52 Design Certification Reviews,” February 19, 2002

12.  IEC 1771, “Nuclear Power Plants Main Control Rooms Verification and Validation of

Design,” (International Electro technical Commission, 1995)

65001.23-05    PROCEDURE COMPLETION

Implementation of this IP is complete when the planned sample of attributes for the specified appendices has been completed.

END

Appendix A:  “Inspection Guide for AP1000 Human Factors Engineering Verification and Validation Activities” 

Revision History Table for IP 65001.23

IP 65001.23 Appendix A

Inspection Guide for AP1000 Human Factors Engineering Verification and Validation Activities

A.01		INSPECTION OBJECTIVE

To confirm by inspection that the combined license (COL) holder (licensee) has implemented Human Factors Engineering (HFE) verification and validation (V&V) activities in accordance with NRC approved implementation plans.  The inspection results will be used to support an NRC finding as to whether the V&V activities meet the acceptance criteria as stated in the AP1000 HFE Inspections, Tests, Analyses and Acceptance Criteria (ITAAC) Design Commitments 3.2.01a, b, c and d

A.02		SAMPLE SIZE

The sample sizes are defined for each inspection activity in the following sections of this procedure. 

A.03		INSPECTION REQUIREMENTS AND GUIDANCE

General Guidance

The ITAAC listed above track completion of the HFE design in accordance with DAC guidance in SECY 092-53. Rather than verifying the as-built HFE design configuration conforms to the approved design (as a typical ITAAC inspection would) this inspection verifies the final design is derived in accordance with the implementation plans the staff approved as part of the design certification.  The implementation plans describe the design process to be used and served as the basis for the staff’s safety evaluation in lieu of a final design. 

The Integrated System Validation (ISV), addressed in ITAAC 3.2.01.c, is a performance based test of the HFE design that demonstrates its effectiveness and is the primary focus of this inspection. Design and corrective action verification activities are also addressed in this inspection because ISV provides an efficient opportunity to independently assess the adequacy of these activities.  In all cases the licensee submits a results summary report documenting the design results and submits that information to support ITAAC closure.  The results of this inspection and the results summary report should be used together to support verification of ITAAC closure. 

The following table provides references to the documents needed to support this inspection.

Issue Date:  12/22/14	AppA-1	65001.23

These documents are proprietary so the inspection requirements identified in the following sections are generalized with references to the proprietary documents where the specific acceptance criteria are described.  This inspection procedure does not contain proprietary information. 

The general methodology for this inspection activity is to directly observe and inspect samples of activities throughout the ISV process in order to obtain reasonable assurance that the licensee conducts the ISV in accordance with the commitments in the implementation plan. It is expected that this inspection will: 

· assess the adequacy of the scope and fidelity of the ISV facility,

· verify that the test crews are representative of AP1000 operating crews,

· verify test crew and observer training has been completed,

· verify the test scenarios meet the implementation plan criteria, 

· verify pilot testing has been completed, and

· observe the ISV performance tests and licensee identification of human error deficiencies (HEDs).  

The ISV is the primary focus of this inspection.  Its secondary purpose is to verify licensee’s actions on the following supporting ITAAC:

· verify Design specifications are implemented (3.2.01a),

· verify Task Analysis specifications are implemented (3.2.01b), and

· review corrective action documentation for closed deficiency reports (3.2.01d).

The composition of inspection teams for each of these elements may vary according to the element’s focus but collectively should include individuals with the following technical competencies: human factors engineering, licensed operator training, and operator licensing.  It is recommended that the inspection team include individuals with integrated nuclear power plant operations experience.

PART 1: Inspection Requirements

A.03.01    ISV Facility

a. Verify that the ISV facility meets the main control room (MCR) scope, fidelity, and capabilities described in APP-OCS-GEH-320, Sections 2.1 through 2.3.

b. Verify significant differences between the ISV facility and AP1000 reference design documents are documented (APP-OCS-GEH-320, Section 2.1).

c. Verify that Remote Shutdown Workstation (RSW) simulation meets the scope, fidelity, and capabilities criteria stated in APP-OCS-GEH-320, Section 2.1.

d. Verify the ISV facility satisfies general requirements of Section 3 and 4 of ANSI/ANS-3.5 (1998 or 2009, as applicable) "Nuclear Power Plant Simulators for Use in Operator Training and Examination" (APP-OCS-GEH-320, Section 2). These 

Issue Date:  12/22/14	AppA-2	65001.23

requirements generally verify the simulator accurately models the control room design.  The control room design is reflected in design specifications and it is the design specification documented in APP-OCS-J1-002 that are used to provide specific criteria for this step in the inspection in accordance with the following table.  The findings from this activity contribute to the staff’s conclusion on task support and design verification ITAAC.

Guidance: Each section listed below contains mandatory guidelines and recommendations.  *The inspection sample is drawn only from the mandatory guidelines and should be limited to those guidelines that can be inspected under static conditions*.  Guidelines that are better demonstrated during simulator operation are embedded in the acceptance criteria for each scenario and are better evaluated by observing how the licensee evaluates the acceptance criteria. For example: “R1.360: The VDU hardware and software should be reliable.” If any HSI locks-up or crashes occur during the scenario it should cause the scenario to fail. The licensee would document and resolve the failure.

Guidance:  Inspection of the ISV facility should be conducted in advance of observation of the ISV performance tests to allow for resolution of any findings prior to the licensee’s conduct of the ISV performance tests.  The objective for inspecting the ISV facility is to verify that the ISV facility is a valid representation of the AP1000 MCR and RSW designs (i.e., any differences between the ISV facility and the AP1000 MCR and RSW designs will not prevent, either individually or collectively, a reasonable assurance determination that performance observed in the ISV facility is predictive of future integrated system performance of an AP1000 MCR or AP1000 RSW).

Issue Date:  12/22/14	AppA-3	65001.23

A.03.02    ISV Test Design, Test Procedures, and Performance Measures

a. Verify a detailed trial assignment and trial order has been established and that it addresses the scheduling factors stated in APP-OCS-GEH-320, Section 3.2. 

b. Verify an ISV procedure (Scenario Package) exists for each scenario and contains the information stated in APP-OCS-GEH-320, Section 5.2.1.

c. Verify Questionnaires are available and contain the information described in APP-OCS-GEH-320, Section 6.2. 

d. Verify Observer Guides are available and contain the information described in APP-OCS-GEH-320, Section 6.2. Key milestones should be identified to provide observers with preview and context information and to support the identification of errors of omission.

e. Verify acceptance criteria are specified for each scenario and address the guidance in APP-OCS-GEH-320, Sections 6.1 and 6.3.

Guidance:  For test materials and performance measures that are scenario-specific (e.g., observer checklists), it is not necessary to inspect each scenario-specific instance.  Rather, the inspector may inspect a sample of approximating 20 percent of each scenario-specific item.  The inspector should examine the rationale for any decisions to relax acceptance criteria based on pilot test results to verify that the decisions are consistent with the objectives of the ISV.

A.03.03    Selection and Training of Test Crew Members

a. Verify that the individuals who will participate as test crew members in the ISV test trials meet the training and experience criteria stated in APP-OCS-GEH-320, Section 4.1. 

b. Verify the ISV test scenarios differ from the scenarios used in crew training (e.g., the initial plant conditions or the timing, magnitude, or combination of failures differ from those of the training scenarios).

A.03.04    ISV Scenario Test Set****

a. Verify that all operational conditions identified in the Operational Condition sample (APP-OCS-GEH-320, Section 5.1.1) are addressed within the test scenarios.

b. Verify that the test scenarios include complications as stated in APP-OCS-GEH-320, Section 5.1.3, and that these complications achieve the goals outlined in this section. 

Guidance:  These activities may be conducted as an in-office inspection of documentation as part of preparation for an on-site inspection of the ISV facility or during the onsite inspection.

Issue Date:  12/22/14	AppA-4	65001.23

A.03.05    Pilot Testing

a. Verify pilot testing was completed and confirmed the items listed in APP-OCS-GEH-320, Section 3.3. Verify deficiencies were documented and resolved. 

A.03.06    ISV Performance Tests

a. Verify a minimum of four test crews participate in the test trials and the crews are used to accomplish the objectives listed in APP-OCS-GEH-320, Section 4.1.2 (last paragraph).

b. Verify the observation teams conform to the guidance in APP-OCS-GEH-320, Section 4.2.

c. Verify the ISV staff conforms to the guidance in APP-OCS-GEH-320, Section 4.3.

d. Verify the ISV coordinator produces a trial log for each trial and that the trial log conforms to APP-OCS-GEH-320, Section 5.2.4 (last paragraph). 

e. Through inspector observation of a sample of 8 or more ISV test trials, verify the following:

1. Test personnel****follow the test protocol.

2. Test crews respond to the test scenarios similarly to how a crew at an operating plant would respond.  No scenario is performed twice by a single crew. 

3. Observers provide a critical review and complete observation checklists completely and accurately.

4. Debriefings are conducted at the end of each trial and week of testing.

5. Workload rating scales are administered.

6. Questionnaires are administered.

7. Plant performance recording is implemented and provides a time-stamped record of plant behavior and operating performance over the course of each observed scenario.

8. Video and audio recordings are implemented and capture the operators’ primary and secondary task inputs.

9. Any problems with implementation of the test protocol or data collection are identified and resolved.

10. The data technician evaluates trial results daily. 

Guidance:  The inspector should select a sample of ISV test trials that allow observation of multiple test crews and data collection personnel.  The sample should also include ISV test scenarios that represent a range of operational conditions and require operator interaction with a range of personnel, procedures, interfaces, and plant systems.  Careful consideration should be given to the number and competencies of the inspectors conducting the observations so as to ensure adequate observation of the performance of each member of the test crew and to assess appropriate licensee identification and prioritization of any HEDs.  The inspectors should independently apply the observer checklists for performance test trials included in the inspection sample.

f. Trial replications are performed in accordance with APP-OCS-GEH-320, Section 3.1

Based on the above activities verify the following objectives were met:

Issue Date:  12/22/14	AppA-5	65001.23

1. The role of plant personnel, i.e., that the allocation of functions to human and automatic aspects of the integrated system are appropriate and takes advantage of human strengths and avoid allocating functions that would be negatively affected by human limitations.

2. Shift staffing, assignment of tasks to crew members and crew coordination (both within the control room as well as between the control room and local control stations and support centers) is acceptable.  This should include validation of the nominal shift levels, minimal shift levels, and shift turnover.

3. For each human function, the design provides adequate alerting, information, control, and feedback capability for human functions to be performed under normal plant evolutions, transients, design basis accidents, and selected risk-significant events that are beyond design basis.

4. Specific personnel tasks can be accomplished within the time and performance criteria, with a high degree of operating crew situation awareness, and within acceptable workload levels that provide a balance between vigilance and operator burden. 

5. The operator interfaces minimize operator error and provide for error detection and recovery capability when errors occur.

6. The functional requirements are met for the major HSI features, e.g., group-view display, alarm system, safety parameter display system (SPDS) function, general display system, procedures, controls, communication systems, and emergency operating procedure (EOP)-related local control stations.

7. The crew can make effective transitions between the HSI features in the accomplishment of their tasks and that interface management tasks such as display configuration and navigation are not a distraction or undue burden.

8. The integrated system performance is tolerant of failures of individual HSI features.

9. Aspects of the integrated system (including staffing, communications, and training) that may negatively impact integrated system performance are identified.

Guidance:  This list constitutes the ISV objectives.  The sample of test trials may not be sufficiently large to reach a conclusion on all the objectives. If this is the case, document where information was insufficient.

A.03.07	Processing of Results and Resolving HEDs

a. Verify that any observed test trials not meeting pass/fail acceptance criteria are identified and documented in a priority 1 deficiency report.

b. Verify that any observed test trials not meeting diagnostic acceptance criteria are identified in a deficiency report of the proper priority in accordance with APP-OCS-GEH-420, Section 2.2.

c. Verify all deficiency reports are tracked in the HFE Tracking System.

d. The findings from this activity contribute to the staff’s conclusion on the discrepancy resolution ITAAC.  Through a sample of 10-20 closed deficiency reports, verify the following:

Issue Date:  12/22/14	AppA-6	65001.23

1. Priority assignment is appropriate.  (APP-OCS-GEH-420, Section 2.2)

2. Trend codes are appropriate. (APP-OCS-GEH-420, Section 2.3)

3. Verify the resolution appropriately addresses the problem. If a deficiency is justified rather than resolved, verify the justification provides an acceptable basis for accepting the deficiency. 

PART 2: Inspection Requirements

Part 2 of this inspection is dependent on the compilation and analysis of data from the test trials which will not be available until the test trials are completed.  If interim results are available the inspection team completing Part 1 should verify these results are consistent with the inspection guidance below.  Otherwise, Part 2 of this inspection is completed as part of ITAAC closure.

A.03.07    Data Analysis and Interpretation

a. By reviewing deficiency reports identifying crosscutting and/or generic issues, verify that the analysis and interpretation of test results conforms to APP-OCS-GEH-320, Section 7.2

b. Verify retest requirements have been applied in accordance with direction in accordance with APP-OCS-GEH-320, Section 7.3 and APP-OCS-GEH-420, Section 2.8.

REFERENCES

1. APP-OCS-GEH-320, “AP1000 Human Factors Engineering Integrated System Validation Plan,” Westinghouse Electric Company LLC.

2. APP-OCS-GEH-321, “AP1000 Human Factors Engineering Integrated System Validation Scenario Information,” Westinghouse Electric Company LLC.

3.   ANSI/ANS 3.5 – 1998: Nuclear Power Plant Simulators for Use in Operator Training and Examination (American National Standards Institute, 1998)

4.   ANSI/ANS 3.5 – 2009: Nuclear Power Plant Simulators for Use in Operator Training and Examination (American National Standards Institute, 2009)

5. APP-OCS-GEH-420, “AP1000 Human Engineering Discrepancy Resolution Process,” Westinghouse Electric Company LLC.

6. APP-OCS-J1-002, “AP1000 Human System Interface Design Guidelines,” Westinghouse Electric Company LLC. 

Issue Date:  12/22/14	AppA-7	65001.23

Attachment 1 - Revision History Table for IP 65001.23 – Appendix 1

INSPECTION OF HUMAN FACTORS ENGINEERING INTEGRATED SYSTEM VALIDATION ITAAC

Issue Date:  12/22/14	Att1-1	65001.23

Issue Date:  12/22/14	Att1-2	65001.23