U.S. DEPARTMENT OF COMMERCE

Donald L. Evans, Secretary

TECHNOLOGY ADMINISTRATION

Phillip J. Bond, Under Secretary of Commerce for Technology

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

Arden L. Bement, Jr., Director

Comparison of ISO/IEC 17025 with the

NUPIC Audit Checklist

C. Douglas Faison

Nationa Voluntary Laboratory Accreditation Program

Standard Services Division

Technology Services

May 2003

NISTIR 6989
NISTIR 6989 May 2003 ii

ACKNOWLEDGMENTS

The author wishes to thank Clinton Eldridge, of the Pacific Gas & Electric Company, for his input and

guidance during the development of this comparison.

The author also wishes to thank Jeffrey Horlick, of NVLAP, for his critical review. His comments

added clarity to the document making it much easier to read.
NISTIR 6989 May 2003 iii

Table of Contents

Introduction ..........................................................1

Basis for Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

The Comparison

Section I - Training/Qualifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Section II - Calibration System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Section III - Contract Requirements ......................................9

Section IV - Adequacy of Measurement Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Section V - Calibration Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Section VI - Environmental Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Section VII - Intervals of Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Section VIII - Calibration Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Section IX - Calibration Traceability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Section X - Subcontractor Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Section XI - Storage and Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Section XII - Out of Tolerance/Corrective Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Section XIII - Calibration System Adequacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Section XIV - Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Appendix A, ISO/IEC 17025 Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
iv

(blank page)
NISTIR 6989 May 2003 1

Introduction

A meeting was held on August 5, 2002, in San Diego, Ca, during the National Conference of

Standards Laboratories International (NCSLI) Conference and Symposium, to discuss Nuclear

Regulatory Commission (NRC) recognition of the use of accredited commercial calibration laboratories

as suppliers of commercial grade calibration services to regulated utilities. In attendance were:

Name Affiliation

Dick Pettit Sandia National Laboratories

Clint Eldridge Pacific Gas & Electric

Larry Nielsen Southern California Edison

Jack Ferris Sleeping Bear Metrology

John Ragsdale Tennessee Valley Authority (TVA)

Roxanne Robinson American Association for Laboratory Accreditation (A2LA)

(National Cooperation for Laboratory Accreditation (NACLA) President)

Tony Anderson Guildline Instruments (NACLA Board member)

Mary Saunders National Institute of Standards and Technology (NIST)

(NIST representative to NACLA Board)

Doug Faison NIST/National Voluntary Laboratory Accreditation Program (NVLAP)

The purpose of the meeting was to map out a strategy for approaching the NRC concerning the issue of

calibration laboratory accreditation and the National Cooperation for Laboratory Accreditation

(NACLA). The intent is to gain NRC endorsement of laboratory accreditation based on internationally

accepted standards as a means of qualifying calibration service providers as suppliers to nuclear power

facilities.

One of the group’s defined tasks was to develop a gap analysis between the internationally accepted

requirements for laboratory accreditation and NRC requirements. Copies of several NRC documents

were obtained and, after a review of those documents, and on the advice of industry experts, it was

decided that the comparison should be conducted using the audit requirements of the Nuclear Utilities

Procurement Issues Committee (NUPIC), which most accurately reflect the requirements of 10CFR50,

Appendix B, Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants.

This document is, therefore, a comparison of NRC requirements, as defined by the NUPIC

Commercial Grade Survey Checklist for Calibration Services, with the internationally accepted

requirements for laboratory accreditation, contained in ISO/IEC 17025:1999, General requirements

for the competence of testing and calibration laboratories. This document does not specifically

discuss the process of laboratory accreditation and/or the process of peer evaluations of laboratory

accreditation bodies for the purpose of mutual recognition. The left column gives the verbatim text

from the NUPIC Checklist, the center column is the relevant text from 17025, and the right column

provides commentary on the comparison between the two.
NISTIR 6989 May 2003 2

Basis for the Comparison

The NUPIC Checklist is a set of 18 Methods of Verification spread over fourteen Sections of the

checklist as follows:

Section I - Training/Qualifications

Section II - Calibration System Description

Section III - Contract Requirements

Section IV - Adequacy of Measurement Standards

Section V - Calibration Procedures

Section VI - Environmental Controls

Section VII - Intervals of Calibration

Section VIII - Calibration Status

Section IX - Calibration Traceability

Section X - Subcontractor Calibration

Section XI - Storage and Handling

Section XII - Out of Tolerance/Corrective Action

Section XIII - Calibration System Adequacy

Section XIV - Records

The language of the Methods of Verification appears to be taken from, or is at least very similar to, the

requirements of ANSI/NCSL Z540-1-1994, the American National Standard for Calibration -

Calibration Laboratories and Measuring and Test Equipment - General Requirements. For each

of the 18 Methods of Verification, there is reference to the relevant Z540 requirements section or

clause(s).

For comparison purposes, only sections 4 and 5 of 17025 are relevant since Sections 1 through 3

contain the Scope of the Standard, References, and Definitions, but no requirements. However, clauses

1.1 and 1.2 of the Scope provide useful guidance as to the applicability of the standard as follows:

17025 clause 1.1 “This International Standard specifies the general requirements for the

competence to carry out tests and/or calibrations, including sampling. It covers

testing and calibration performed using standard methods, non-standard

methods, and laboratory-developed methods.”

17025 clause 1.2 “This International Standard is applicable to all organizations performing tests

and/or calibrations. These include, for example, first-, second- and third-party

laboratories, and laboratories where testing and/or calibration forms part of

inspection and product certification.”

Based on the way the NUPIC Methods of Verification are expressed, it is difficult in some cases to

determine what would be considered as acceptable compliance. Without further guidance, compliance

may be subject to the interpretation of the requirements by individual auditors. As can be seen in the

comparison, ISO/IEC 17025, as implemented through the process of laboratory accreditation, is very

specific concerning compliance to the standard.

As previously stated, the focus of this effort is on gaining NRC acceptance of accredited laboratories as

suppliers of commercial grade calibration services only. Thus, the requirements of 10CFR21; Nuclear

Regulatory Commission, Reporting of Defects and Noncompliance, are not relevant to this

comparison.

Conclusions
NISTIR 6989 May 2003 3

In the final analysis, the comparison shows that 17025 adequately addresses all but two administrative

issues, namely:

1. NUPIC clause 14.1.c.7 “The calibration certificate/report shall include identification of the

laboratory equipment/standards used.” 17025 does not require this, but

does require that records be kept.

2. NUPIC clause 14.1.c.12 “The calibration certificate/report shall include as-found and as-left

data.” 17025 only requires as-found data to be reported when the

device under test is adjusted/repaired.

One additional issue requires explanation. Clause 4.1 of the NUPIC checklist states, in part, “Well

defined and documented measurement assurance techniques or uncertainty analyses may be used to

verify the adequacy of the measurement process. If such techniques are not used, the collective

uncertainty of the measurement standards shall not exceed 25% of the acceptable tolerance for each

characteristic being calibrated.” This is typically referred to as the four-to-one ratio. The NUPIC

Checklist does not specifically require the four-to-one ratio but does allow its use if an uncertainty

analysis is not available. 17025 requires an uncertainty analysis to support all measurement results,

therefore the four-to-one ratio rule is not applicable. 

However, clause 12.6 of Standard Review Plan (SRP) Section 17.1, Quality Assurance During the

Design and Construction Phases Review Responsibilities, states “Calibration of this equipment

should be against standards that have an accuracy of at least four times the required accuracy of the

equipment being calibrated or, when this is not possible, have an accuracy that assures the equipment

being calibrated will be within required tolerance and that the basis of acceptance is documented and

authorized by responsible management. The management authorized to perform this function is

identified.” Although the SRP is not a requirements document and uses the term “should” instead of

“shall”, the four-to-one ratio has become something of a default requirement and industry experts

believe this issue should be considered in this comparison.

Each of the three issues noted above can easily be addressed by requiring the utilities to add language to

their purchase orders for commercial grade calibration services. For example, to address the four-toone ratio, the following requirement could be added. “the collective expanded uncertainty of the

measurement standards must not exceed 25% of the acceptable tolerance for each characteristic being

calibrated." Adding this statement to the purchase order would ensure that an uncertainty statement be

provided, as required by 17025, and that the four-to-one ratio required by SRP 17.1 be maintained or

exceeded. Similar statements could be added to the purchase order to require that standards used in the

calibration process and all as-found and as-left data be reported in the Calibration Certificate/Report.

It should also be noted that 17025 contains many requirements, both management and technical, that are

not addressed by the NUPIC checklist and are not included in the comparison. Laboratories accredited

to 17025 have more thoroughly addressed those issues necessary to assure competence, capability, and

traceability of measurement results.
4

(blank page)
NISTIR 6989 5 

Comparison of the NUPIC Commercial Grade Survey Checklist for Calibration Serviceswith ISO/IEC 17025: General requirements for the competence of testing and calibration laboratories.

From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 CommentsSection I - Training/Qualifications1.1 Verify personnel have sufficienteducation, training, technicalknowledge, and experience for theirassigned functions.Review personnel qualificationrecords, which provide the basis foran adequate training and qualificationprogram.

5.2.1 The laboratory management shall ensure the competenceof all who operate specific equipment, perform tests and/orcalibrations, evaluate results, and sign test reports and calibrationcertificates. When using staff who are undergoing training,appropriate supervision shall be provided. Personnel performingspecific tasks shall be qualified on the basis of appropriateeducation, training, experience and/or demonstrated skills, asrequired.5.2.3 The laboratory shall use personnel who are employed by,or under contract to, the laboratory. Where contracted andadditional technical and key support personnel are used, thelaboratory shall ensure that such personnel are supervised andcompetent and that they work in accordance with the laboratory'squality system.5.2.5 The management shall authorize specific personnel toperform particular types of sampling, test and/or calibration, toissue test reports and calibration certificates, to give opinions andinterpretations and to operate particular types of equipment. Thelaboratory shall maintain records of the relevant authorization(s),competence, educational and professional qualifications, training,skills and experience of all technical personnel, includingcontracted personnel. This information shall be readily availableand shall include the date on which authorization and/orcompetence is confirmed.

Section 5.2 of 17025 ithat all personnel are tperform their duties. 17025 identifies manato ensure competence oversight of trainees. 17025 requires that althose under contract, quality system.17025 requires that mresponsibility for authindividuals to perform appropriate records be 
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 6 

(continued)Assess continuing training ofcalibration personnel.

5.2.2 The management of the laboratory shall formulate the

goals with respect to the education, training and skills of thelaboratory personnel. The laboratory shall have a policy andprocedures for identifying training needs and providing trainingof personnel. The training program shall be relevant to thepresent and anticipated tasks of the laboratory.

17025 specifically reqpolicies and proceduretraining needs and tharesponsible.

Section II - Calibration SystemDescription2.1 Verify that the supplier hasestablished and documented a qualitysystem appropriate to the type, rangeand volume of calibration activitiesperformed.

4.2.1 The laboratory shall establish, implement and maintain aquality system appropriate to the scope of its activities. Thelaboratory shall document its policies, systems, programs,procedures and instructions to the extent necessary to assure thequality of the test and/or calibration results.

17025 specifies in detaa documented quality Section 4 Managemenrelevant.

a. Are the calibration supplier’spolicies and objectives defined?

4.2.2 The laboratory’s quality system policies and objectivesshall be defined in a quality manual (however named). The

overall objectives shall be documented in a quality policystatement. The quality policy statement shall be issued under theauthority of the chief executive. It shall include at least thefollowing:a) the laboratory management's commitment to good

professional practice and to the quality of its testing and

calibration in servicing its clients;

b) the management’s statement of the laboratory's standard

of service;

c) the objectives of the quality system;d) a requirement that all personnel concerned with testing

and calibration activities within the laboratoryfamiliarize themselves with the quality documentationand implement the policies and procedures in theirwork; and 

e) the laboratory management's commitment to compliance

with this International Standard.

17025, section 4.2 Quthat policies and objecthey be documented in statement, and details the policy statement. 
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 7 

b. Is the quality documentationavailable for use by calibrationpersonnel?

4.2.1 ....The system’s documentation shall be communicated to,understood by, available to, and implemented by the appropriate

personnel.

17025 requires that thnot only be available fbe communicated to aappropriate personnel

c. Is there an organization chartavailable? 

4.1.5 The laboratory shalle) define the organization and management structure of the

laboratory, its place in any parent organization, and therelationships between quality management, technicaloperations and support services;

f) specify the responsibility, authority and

interrelationships of all personnel who manage, performor verify work affecting the quality of the tests and/orcalibrations;

4.2.4 The roles and responsibilities of technical managementand the quality manager, including their responsibility forensuring compliance with this standard, shall be defined in the

quality manual.

17025 requires that thdefined and that interrpersonnel also be spec

d. Does the quality system documentprovide for: scope of lab activities,personnel training, procedure control,maintenance of standards,identification and control ofdiscrepancies, internal audits, andreports/records.

4.2.3 The quality manual shall include or make reference to thesupporting procedures including technical procedures. It shalloutline the structure of the documentation used in the quality

system.

This requirement appefrom Z540-1, section the quality manual and 

shall also include... Tdeleted in 17025 becauincorrectly developing only the listed elemenQuality System, requi

the standard be addresincluding those specifichecklist.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 8 

e. Does quality system documentaddress 10CFR50 Appendix B and10CFR21? Does supplier acceptcustomer orders with 10CFR21invoked? (for information purposesonly)

Not a specific requirement of 17025. However, the relevantclauses are:4.2.1 The laboratory shall establish, implement and maintain aquality system appropriate to the scope of its activities. 4.4.1 The laboratory shall establish and maintain procedures for

the review of requests, tenders and contracts. The policies andprocedures for these reviews leading to a contract for testingand/or calibration shall ensure that:a) the requirements, including the methods to be used, are

adequately defined, documented and understood (see5.4.2);

b) the laboratory has the capability and resources to meet

the requirements;

Only commercial gradare being considered. not applicable. The NUPIC checklist the requirements of 10Therefore, it is 17025 relevant. 
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 9 

Section III - ContractRequirements3.1 Verify that deviations/exceptionsto customer purchase orders aredocumented and controlled:a. Exceptions to purchase orderdiscussed/resolved at order entry,prior to work

4.4.1 The laboratory shall establish and maintain procedures forthe review of requests, tenders and contracts. The policies andprocedures for these reviews leading to a contract for testingand/or calibration shall ensure that:a) the requirements, including the methods to be used, are

adequately defined, documented and understood (see5.4.2);

b) the laboratory has the capability and resources to meet

the requirements;

c) the appropriate test and/or calibration method is selected

and capable of meeting the clients' requirements (see5.4.2).

Any differences between the request or tender and the contract

shall be resolved before any work commences. Each contractshall be acceptable both to the laboratory and the client.4.4.2 Records of reviews, including any significant changes,shall be maintained. Records shall also be maintained of pertinentdiscussions with a client relating to the client's requirements orthe results of the work during the period of execution of thecontract.4.4.3 The review shall also cover any work that is subcontractedby the laboratory.4.4.4 The client shall be informed of any deviation from thecontract.4.4.5 If a contract needs to be amended after work hascommenced, the same contract review process shall be repeatedand any amendments shall be communicated to all affectedpersonnel.

17025 has added a comdeals with the review contracts which specifnecessary steps to be tkept, etc.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 10 

b. Damaged/defective equipmentnoted at receipt c. Out-of-tolerance condition ofcustomer equipment noted

5.8.3 Upon receipt of the test or calibration item, abnormalities

or departures from normal or specified conditions, as describedin the test or calibration method, shall be recorded. When thereis doubt as to the suitability of an item for test or calibration, orwhen an item does not conform to the description provided, orthe test or calibration required is not specified in sufficient detail,the laboratory shall consult the client for further instructionsbefore proceeding and shall record the discussion.

17025 requirements foexceed those of the NU

Section IV - Adequacy ofMeasurement Standards4.1 Verify the measurementstandards used by the supplier forcalibration of M&TE and othermeasurement standards have theaccuracy, stability, range, andresolution required for the instrumentcalibrated.

5.5.1 The laboratory shall be furnished with all items ofsampling, measurement and test equipment required for thecorrect performance of the tests and/or calibrations (includingsampling, preparation of test and/or calibration items, processingand analysis of test and/or calibration data). In those cases where

the laboratory needs to use equipment outside its permanentcontrol, it shall ensure that the requirements of this standard aremet.5.5.2 Equipment and its software used for testing, calibrationand sampling shall be capable of achieving the accuracy requiredand shall comply with specifications relevant to the tests and/orcalibrations concerned. Calibration programs shall be establishedfor key quantities or values of the instruments where theseproperties have a significant effect on the results. Before beingplaced into service, equipment (including that used for sampling)shall be calibrated or checked to establish that it meets thelaboratory's specification requirements and complies with therelevant standard specifications. It shall be checked and/orcalibrated before use.

17025 does not differerequirements for custothose done in support e.g., on in-house M&of 17025 is to ensure tby the lab are approprincludes all calibrationor through calibration lab, in support of the provided by the lab. Amust be able to show measurements are sciethey are traceable to areferences. This includtraceability of standardused in performance oFurthermore, uncertaiprovided must include to these standards, M&equipment to the extenmeasurement results. 
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 11 

The laboratory shall ensure that themeasurement uncertainties aresufficiently small as (sic) that theadequacy of the measurement is notaffected.

5.4.6.1 A calibration laboratory, or a testing laboratoryperforming its own calibrations, shall have and shall apply aprocedure to estimate the uncertainty of measurement for all

calibrations and types of calibrations.5.10.4.2 When statements of compliance are made, theuncertainty of measurement shall be taken into account.

All parts of the requirfor providing service aneeds. 17025 does nosize of the uncertaintydetermine the adequac

Hence, the uncertainty 

Well defined and documentedmeasurement assurance techniques oruncertainty analyses may be used toverify the adequacy of themeasurement process.

5.4.6.3 When estimating the uncertainty of measurement, alluncertainty components which are of importance in the given

situation shall be taken into account using appropriate methods of

analysis.

17025 requires that unwith environmental faof those factors, and ocontributions from meconsidered.

If such techniques are not used, thecollective uncertainty of themeasurement standards shall notexceed 25% of the acceptabletolerance for each characteristic beingcalibrated.

5.4.6.1 A calibration laboratory, or a testing laboratoryperforming its own calibrations, shall have and shall apply aprocedure to estimate the uncertainty of measurement for all

calibrations and types of calibrations.5.4.6.3 When estimating the uncertainty of measurement, alluncertainty components which are of importance in the givensituation shall be taken into account using appropriate methods of

analysis.

The NUPIC statement well defined and docuassurance techniques oare not available. In 1must be supported by uncertainty analysis, tthe 25% rule. It should be noted that assure traceability of man acceptable alternatithorough uncertainty a“uncertainty of the meincorrect. Uncertaintyassociated with the resor the value of a standused in making the me
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 12 

Section V - Calibration Procedures5.1 Verify procedures are availableand are utilized for calibration ofM&TE and standards and that theyspecify:Standard(s) to be usedEquipment to be usedThe required parameter, range, andaccuracy of the standard

5.4.1 The laboratory shall use appropriate methods andprocedures for all tests and/or calibrations within its scope.These include sampling, handling, transport, storage andpreparation of items to be tested and/or calibrated, and, whereappropriate, an estimation of the measurement uncertainty aswell as statistical techniques for analysis of test and/or calibrationdata.5.4.2 The laboratory shall use test and/or calibration methods,

including methods for sampling, which meet the needs of theclient and which are appropriate for the tests and/or calibrationsit undertakes.5.5.2 Equipment and its software used for testing, calibrationand sampling shall be capable of achieving the accuracy requiredand shall comply with specifications relevant to the tests and/orcalibrations concerned. Calibration programs shall be establishedfor key quantities or values of the instruments where theseproperties have a significant effect on the results. Before beingplaced into service, equipment (including that used for sampling)shall be calibrated or checked to establish that it meets thelaboratory's specification requirements and complies with therelevant standard specifications. It shall be checked and/orcalibrated before use.5.6.1 All equipment used for tests and/or calibrations, includingequipment for subsidiary measurements (e.g., for environmentalconditions) having a significant effect on the accuracy or validityof the result of the test, calibration or sampling shall becalibrated before being put into service. The laboratory shallhave an established program and procedure for the calibration ofits equipment.

Although 17025 is lesrequirements for the cprocedures, sufficient through the requiremeuncertainty analyses to 

and supporting equipmto meet the needs of thAs previously discusseall real or potential infuncertainty of measureaddressed.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 13 

5.2 Verify that when a devicetolerance (range) is not known orspecified and an uncertainty(regression) analysis is performed,adequate procedure controls exist tocontrol sampling of data used for the

analysis. Assess therationale/justification for thesampling plan selected.

5.4.1 The laboratory shall use appropriate methods and

procedures for all tests and/or calibrations within its scope.These include sampling, handling, transport, storage andpreparation of items to be tested and/or calibrated, and, whereappropriate, an estimation of the measurement uncertainty aswell as statistical techniques for analysis of test and/or calibrationdata.5.7.1 The laboratory shall have a sampling plan and proceduresfor sampling when it carries out sampling of substances,materials or products for subsequent testing or calibration. Thesampling plan as well as the sampling procedure shall beavailable at the location where sampling is undertaken. Samplingplans shall, whenever reasonable, be based on appropriatestatistical methods. The sampling process shall address thefactors to be controlled to ensure the validity of the test andcalibration results.5.7.3 The laboratory shall have procedures for recordingrelevant data and operations relating to sampling that forms partof the testing or calibration that is undertaken. These records

shall include the sampling procedure used, the identification of

the sampler, environmental conditions (if relevant) and diagramsor other equivalent means to identify the sampling location asnecessary and, if appropriate, the statistics the sampling

procedures are based upon.

This NUPIC requiremZ540-1, clause 10.5 wsampling is carried oucalibration method, nodata is used for the unreality, all data used fare sample data. Alsomeasurement result is the tolerance of the deknown or specified.The original requiremGuide 25, which was was applicable to both labs. Sampling is a coby test labs where, for a batch are tested, but where the test is destruselected samples are tewas left in Z540-1 for where sampling may bcalibrations. One exaroll of thermocouple wnot tested, but usually are tested: one from ethe middle.The requirement is weshould it be applicable performed where sammethod. 
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 14 

5.3 Verify calculations and datatransfers are subject to appropriatechecks.

5.4.7.1 Calculations and data transfers shall be subject toappropriate checks in a systematic manner.

17025 requires that cha systematic manner.

5.4 Verify when computers orautomated equipment are used for thecapture, processing, manipulation,recording, reporting, storage, orretrieval of calibration data, thelaboratory ensures that:a. Computer software is

documented and adequatefor use;

b. Procedures are established

and implemented forprotecting the integrity andsecurity of data;

c. Computer and automated

equipment is maintained toensure proper functioningand provided with theenvironmental and operatingconditions necessary tomaintain the integrity ofcalibration data.

5.4.7.2 When computers or automated equipment are used forthe acquisition, processing, recording, reporting, storage orretrieval of test or calibration data, the laboratory shall ensurethat:a. computer software developed by the user is documented

in sufficient detail and is suitably validated as beingadequate for use;

b. procedures are established and implemented for

protecting the data; such procedures shall include, butnot be limited to, integrity and confidentiality of dataentry or collection, data storage, data transmission anddata processing;

c. computers and automated equipment are maintained to

ensure proper functioning and are provided with theenvironmental and operating conditions necessary tomaintain the integrity of test and calibration data.

17025 requirements arare an improvement orequirements.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 15 

Section VI - EnvironmentalControls6.1 Verify M&TE and standards arecalibrated and utilized in anenvironment controlled to the extentnecessary to assure continuedmeasurements of required accuracy(i.e., humidity, temperature, EMI,line voltage, sound/vibration levels,and cleanliness).

5.3.1 Laboratory facilities for testing and/or calibration,including but not limited to energy sources, lighting andenvironmental conditions, shall be such as to facilitate correct

performance of the tests and/or calibrations.The laboratory shall ensure that the environmental conditions donot invalidate the results or adversely affect the required qualityof any measurement. Particular care shall be taken whensampling and tests and/or calibrations are undertaken at sitesother than a permanent laboratory facility. The technicalrequirements for accommodation and environmental conditionsthat can affect the results of tests and calibrations shall bedocumented.5.3.2 The laboratory shall monitor, control and recordenvironmental conditions as required by the relevantspecifications, methods and procedures or where they influencethe quality of the results. Due attention shall be paid, forexample, to biological sterility, dust, electromagneticdisturbances, radiation, humidity, electrical supply, temperature,and sound and vibration levels, as appropriate to the technicalactivities concerned. Tests and calibrations shall be stopped whenthe environmental conditions jeopardize the results of the testsand/or calibrations.5.3.3 There shall be effective separation between neighboringareas in which there are incompatible activities. Measures shallbe taken to prevent cross-contamination.5.3.4 Access to and use of areas affecting the quality of the testsand/or calibrations shall be controlled. The laboratory shalldetermine the extent of control based on its particularcircumstances. 5.3.5 Measures shall be taken to ensure good housekeeping inthe laboratory. Special procedures shall be prepared wherenecessary.

17025 specifically idedocumentation of condresults and requires thcalibrations are to be senvironmental conditioresults. 17025 also reqon the results of measenvironmental factors uncertainty analyses.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 16 

Section VII - Intervals ofCalibration7.1 Verify the supplier has a recallsystem established to assure M&TEand standards are calibrated atestablished intervals and maintainedto assure acceptable accuracy andreliability.

5.6.1 All equipment used for tests and/or calibrations, including

equipment for subsidiary measurements (e.g., for environmentalconditions) having a significant effect on the accuracy or validityof the result of the test, calibration or sampling shall becalibrated before being put into service. The laboratory shallhave an established program and procedure for the calibration ofits equipment.

While 17025 does not does require that the lprogram and procedursetting intervals in pla

Section VIII - Calibration Status8.1 a. Verify M&TE and standardsare uniquely identified and labeled toindicate calibration status (minimumcalibration date and calibration duedate). Where impractical to label theitem, the vendor shall provide othersuitable means to identify the statusof them.b. Items that are not calibrated totheir full range, or have otherlimitations, shall be identified to thatcondition.c. Also verify that the supplier hascontrols in place to apply tamperresistant seals to operator accessiblecontrols or adjustments, which ifmoved, would affect the calibration. The calibration system shall provideinstructions for use of the seals anddisposition on items whose seals arebroken.

5.5.8 Whenever practicable, all equipment under the controlof the laboratory and requiring calibration shall be labeled, coded

or otherwise identified to indicate the status of calibration,including the date when last calibrated and the date or expiration

criteria when recalibration is due.5.5.12 Test and calibration equipment, including both hardwareand software, shall be safeguarded from adjustments whichwould invalidate the test and/or calibration results.

17025 requires that alor otherwise identified status and its fitness fodetermined. This inclulimitations, etc.17025 does not requirrequire that appropriatemployed.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 17 

Section IX - CalibrationTraceability9.1 Verify that M&TE, calibrationstandards and reference materials are:Traceable to national, international,or intrinsic standards whereavailable.Supported by certificates, reports, ordata sheets

5.6.1 General - All equipment used for tests and/orcalibrations, including equipment for subsidiary measurements(e.g., for environmental conditions) having a significant effect onthe accuracy or validity of the result of the test, calibration orsampling shall be calibrated before being put into service. Thelaboratory shall have an established program and procedure for

the calibration of its equipment.5.6.2.1.1 For calibration laboratories, the program forcalibration of equipment shall be designed and operated so as toensure that calibrations and measurements made by thelaboratory are traceable to the International System of Units (SI)(Système international d'unités).

A calibration laboratory establishes traceability of its ownmeasurement standards and measuring instruments to the SI bymeans of an unbroken chain of calibrations or comparisonslinking them to relevant primary standards of the SI units ofmeasurement. The link to SI units may be achieved by referenceto national measurement standards. National measurementstandards may be primary standards, which are primaryrealizations of the SI units or agreed representations of SI unitsbased on fundamental physical constants, or they may besecondary standards which are standards calibrated by anothernational metrology institute. When using external calibrationservices, traceability of measurement shall be assured by the useof calibration services from laboratories that can demonstratecompetence, measurement capability and traceability. Thecalibration certificates issued by these laboratories shall containthe measurement results, including the measurement uncertaintyand/or a statement of compliance with an identified metrologicalspecification (see also 5.10.4.2).

17025 requires that altraceable to stated refe

definition of the term in the International VoGeneral Terms in Met6.10) and as adopted bFurthermore, all recogbodies (signatories to and/or NACLA MRApolicies on traceability Policy and Guidance dof traceability is fundaaccreditation process. 17025 specifies the actraceability, requires tlabs be able to demonspecifies what, at a mincluded in calibration Furthermore, all accrerequirements to the efservices required to suservices, must come finstitutes or from accr
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 18 

Where traceability to national,international, or intrinsic standards ofmeasurement is not available,traceability requirements may besatisfied by:a) participation in a suitableprogram of inter-laboratorycomparisons or proficiency testingb) internationally acceptedstandards in the field concerned.c) suitable reference materialsd) ratio or reciprocity typemeasurementse) mutual consent standardsagreed upon by all parties concerned

5.6.2.1.2 There are certain calibrations that currently cannot bestrictly made in SI units. In these cases calibration shall provide

confidence in measurements by establishing traceability toappropriate measurement standards such as:

S the use of certified reference materials provided by a

competent supplier to give a reliable physical orchemical characterization of a material;

S the use of specified methods and/or consensus standards

that are clearly described and agreed by all partiesconcerned.

Participation in a suitable program of interlaboratorycomparisons is required where possible.

17025 has provisions traceability to the SI is cases clients are to be service provider labs atheir results with othemeasurements.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 19 

Section X - SubcontractorCalibration10.1 Verify the supplier assures thatsub-contractors providing calibrationservices for their own M&TE andstandards are capable of performingthe required services in accordancewith the documented program (ref.Checklist Item 2.1). Assessestablished controls for subcontractorcapability determined throughaudits/surveys, source surveillance,history, etc.a) customer notification requiredwhen calibration is subcontracted.b) subcontractor capabilitydetermined through audits/surveys,source surveillance, history, etc.

4.6.1 The laboratory shall have a policy and procedure(s) for theselection and purchasing of services and supplies it uses that

affect the quality of the tests and/or calibrations. Procedures shallexist for the purchase, reception and storage of reagents andlaboratory consumable materials relevant for the tests andcalibrations.4.6.2 The laboratory shall ensure that purchased supplies andreagents and consumable materials that affect the quality of testsand/or calibrations are not used until they have been inspected orotherwise verified as complying with standard specifications orrequirements defined in the methods for the tests and/orcalibrations concerned. These services and supplies used shallcomply with specified requirements. Records of actions taken tocheck compliance shall be maintained.4.6.3 Purchasing documents for items affecting the quality oflaboratory output shall contain data describing the services andsupplies ordered. These purchasing documents shall be reviewedand approved for technical content prior to release.4.6.4 The laboratory shall evaluate suppliers of criticalconsumables, supplies and services which affect the quality oftesting and calibration, and shall maintain records of theseevaluations and list those approved.

Clause 10.1 of the NUbe a misinterpretation subcontracting as it ap17025. The requiremapply when the laborasubcontract the service customers. When this rules that must be follWhen a laboratory has M&TE calibrated by adoes not fall under subpurchasing of serviceslaboratories purchase other labs to support ttheir customers. Therto notify the customer There are well defined purchasing of services 17025. 
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 20 

10.2 Where calibration services forcustomer M&TE/standards are subcontracted, access controls for:a) customer notification/approvalrequiredb) customer requirements imposedon sub-contractorb) subcontractor capability to meetapplicable requirements

4.5.1 When a laboratory subcontracts work whether because ofunforeseen reasons (e.g., workload, need for further expertise or

temporary incapacity) or on a continuing basis (e.g., throughpermanent subcontracting, agency or franchising arrangements),this work shall be placed with a competent subcontractor. Acompetent subcontractor is one that, for example, complies withthis standard for the work in question.

4.5.2 The laboratory shall advise the client of the arrangementin writing and, when appropriate, gain the approval of the client,

preferably in writing.4.5.3 The laboratory is responsible to the client for thesubcontractor’s work, except in the case where the client or aregulatory authority specifies which subcontractor is to be used.4.5.4 The laboratory shall maintain a register of allsubcontractors that it uses for tests and/or calibrations and arecord of the evidence of compliance with this standard for thework in question.

17025 requires that suproviders be competennotified in writing, anlaboratory maintains rservices provided to itrequires that a register subcontractors be maievidence of complianc

Section XI - Storage and Handling11.1 Verify M&TE and Standardsare handled, stored, and transportedin accordance with establishedlaboratory procedures to avoiddeterioration or damage, which couldaffect the calibration of theequipment.

5.5.6 The laboratory shall have procedures for safe handling,transport, storage, use and planned maintenance of measuringequipment to ensure proper functioning and in order to preventcontamination or deterioration.

17025 also requires thstandard or M&TE beitem”, the requiremen“Handling of test and apply.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 21 

Section XII - Out ofTolerance/Corrective Action12.1 Verify the supplier’s systemprovides for customer notificationwhen Laboratory’s M&TE/standardsand/or customer’s M&TE/standardsare found to be significantly out-oftolerance. Evaluate the adequacy ofthe definition of significant and theassociated corrective actionmeasures.

4.9.1 The laboratory shall have a policy and procedures that

shall be implemented when any aspect of its testing and/orcalibration work, or the results of this work, do not conform toits own procedures or the agreed requirements of the client. Thepolicy and procedures shall ensure that:a) the responsibilities and authorities for the management

of nonconforming work are designated and actions(including halting of work and withholding of testreports and calibration certificates, as necessary) aredefined and taken when nonconforming work isidentified;

b) an evaluation of the significance of the nonconforming

work is made;

c) corrective actions are taken immediately, together with

any decision about the acceptability of thenonconforming work;

d) where necessary, the client is notified and work is

recalled;

e) the responsibility for authorizing the resumption of work

is defined.

17025 contains requireof the customer when non-conforming workconditions are considework. 17025 also iderecalling non-conformcustomer for recalibra
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 22 

(continued)Also, verify the system provides forthe prevention of inaccuracy bydetection of deficiencies and timely,positive action for their correction.

4.10 Corrective action4.10.1 General - The laboratory shall establish a policy andprocedure and shall designate appropriate authorities forimplementing corrective action when nonconforming work ordepartures from the policies and procedures in the quality system

or technical operations have been identified.4.10.2 Cause analysis - The procedure for corrective actionshall start with an investigation to determine the root cause(s) ofthe problem.4.10.3 Selection and implementation of corrective actions -Where corrective action is needed, the laboratory shall identifypotential corrective actions. It shall select and implement theaction(s) most likely to eliminate the problem and to preventrecurrence.4.10.4 Monitoring of corrective actions - The laboratory shallmonitor the results to ensure that the corrective actions takenhave been effective.4.10.5 Additional audits - Where the identification ofnonconformances or departures casts doubts on the laboratory'scompliance with its own policies and procedures, or on itscompliance with this handbook, the laboratory shall ensure thatthe appropriate areas of activity are audited in accordance with4.13 as soon as possible. 

17025 also requires thmonitored and that whaudits be conducted to of corrective actions t
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 23 

(continued) 4.11 Preventive action

4.11.1 Needed improvements and potential sources ofnonconformances, either technical or concerning the qualitysystem, shall be identified. If preventive action is required,action plans shall be developed, implemented and monitored to

reduce the likelihood of the occurrence of such nonconformancesand to take advantage of the opportunities for improvement.4.11.2 Procedures for preventive actions shall include theinitiation of such actions and application of controls to ensurethat they are effective.

17025 implements an for the laboratory to idof non-conformances preventive action proc
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 24 

Section XIII - Calibration SystemAdequacy13.1 Verify the supplier hasestablished and maintains documentedprocedures to:a) Evaluate the adequacy of thecalibration system through internalaudits or equivalent oversight toverify compliance with establishedrequirements.

4.13.1 The laboratory shall periodically, and in accordance witha predetermined schedule and procedure, conduct internal auditsof its activities to verify that its operations continue to comply

with the requirements of the quality system and this standard.The internal audit program shall address all elements of thequality system, including the testing and/or calibration activities.It is the responsibility of the quality manager to plan andorganize audits as required by the schedule and requested bymanagement. Such audits shall be carried out by trained andqualified personnel who are, wherever resources permit,independent of the activity to be audited. NOTE: The cycle forinternal auditing should normally be completed in one year.4.13.2 When audit findings cast doubt on the effectiveness ofthe operations or on the correctness or validity of thelaboratory's test or calibration results, the laboratory shall taketimely corrective action, and shall notify clients in writing ifinvestigations show that the laboratory results may have beenaffected.4.13.3 The area of activity audited, the audit findings andcorrective actions that arise from them shall be recorded.4.13.4 Follow-up audit activities shall verify and record theimplementation and effectiveness of the corrective action taken.4.14.1 In accordance with a predetermined schedule andprocedure, the laboratory’s executive management shallperiodically conduct a review of the laboratory's quality systemand testing and/or calibration activities to ensure their continuingsuitability and effectiveness, and to introduce necessary changesor improvements.

17025 contains detaileinternal audits and ma17025 also requires thtrained to conduct audthe area being audited
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 25 

b) In addition to internal audits, areinternal checks implemented?: 1. statistical techniques2. proficiency testing/interlaboratory comparisons3. use of certified referencematerials4. use of alternate methods forreplicating measurements

5.9 Assuring the quality of test and calibration resultsThe laboratory shall have quality control procedures formonitoring the validity of tests and calibrations undertaken. Theresulting data shall be recorded in such a way that trends aredetectable and, where practicable, statistical techniques shall beapplied to the reviewing of the results. This monitoring shall beplanned and reviewed and may include, but not be limited to, thefollowing:a) regular use of certified reference materials and/or

internal quality control using secondary referencematerials;

b) participation in interlaboratory comparison or

proficiency-testing programs;

c) replicate tests or calibrations using the same or different

methods;

d) retesting or recalibration of retained items;

e) correlation of results for different characteristics of an

item.

NOTE The selected methods should be appropriate for the typeand volume of the work undertaken.

17025 includes a new “Assuring the quality results.”
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 26 

Section IX - Records14.1 Verify calibration service issupported by records. These recordsshould include, but are not limited to:a. laboratory equipment/referencestandards

5.5.5 Records shall be maintained of each item of equipmentand its software significant to the tests and/or calibrationsperformed. The records shall include at least the following:

17025 includes all reqchecklist plus a requirof the manufacturer’s 

1. name/description of

equipment/standard

2. the identity of the item of equipment and its software;

2. manufacturer’s name,

model, serial number, orother unique identification

3. the manufacturer's name, type identification, and serial

number or other unique identification;

3. laboratory location d. the current location, where appropriate;4. calibration status/date

information

c. Checks that equipment complies with the specification

(see 5.5.2);

f. dates, results and copies of reports and certificates of all

calibrations, adjustments, acceptance criteria, and thedue date of next calibration;

5. history of calibrations,

maintenance, repairs,damage, modifications

g. the maintenance plan, where appropriate, and

maintenance carried out to date;

6. out of tolerance data h. any damage, malfunction, modification or repair to the

equipment.

e. the manufacturer's instructions, if available, or

reference to their location;
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 27 

(14.1) b. calibration personnel1. training courses2. prior education/experience3. qualifications/certifications

5.2.5 The management shall authorize specific personnel toperform particular types of sampling, test and/or calibration, toissue test reports and calibration certificates, to give opinions and

interpretations and to operate particular types of equipment. Thelaboratory shall maintain records of the relevant authorization(s),competence, educational and professional qualifications, training,skills and experience of all technical personnel, includingcontracted personnel. This information shall be readily availableand shall include the date on which authorization and/orcompetence is confirmed.

17025 also requires thauthorizations for cert

c. calibration certificate/report 5.10.2 Test reports and calibration certificates

Each test report or calibration certificate shall include at least thefollowing information, unless the laboratory has valid reasons for

not doing so:

17025 requires that suservices meet the requstandard. The standarinformation be includecertificates/reports. S

1. title, unique identification

number

a) a title (e.g., "Test Report" or "Calibration Certificate");c) unique identification of the test report or calibration

certificate (such as the serial number), and on each pagean identification in order to ensure that the page isrecognized as a part of the test report or calibrationcertificate, and a clear identification of the end of thetest report or calibration certificate;

17025 also requires paend of report identific

2. calibration laboratory

location

b) the name and address of the laboratory, and the location

where the tests and/or calibrations were carried out, ifdifferent from the address of the laboratory;

17025 requires that thwhere the calibration widentified, if different

3. customer

name/address/purchase ordernumber

d) the name and address of the client; 17025 does not requir

number to appear on tlittle relevance to the c

4. unique identification of

calibrated equipment

f) a description of, the condition of, and unambiguous

identification of the item(s) tested or calibrated;

17025 also requires ththe condition of the caprovided.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 28 

5. calibration date g) the date of receipt of the test or calibration item(s)

where this is critical to the validity and application ofthe results, and the date(s) of performance of the test orcalibration;

17025 also requires ththis information is crit

6. calibration procedure used e) identification of the method used; 17025 requires that th

be identified as opposeprocedure.

7. laboratory

equipment/standards used

Not required by 17025 report. 17025 does reinformation be record

records.

8. deviations or out-oftolerance conditions notation i) the test or calibration results with, where appropriate, the units of measurement; and

5.10.4.2 The calibration certificate shall relate only to quantitiesand the results of functional tests. If a statement of compliancewith a specification is made, this shall identify which clauses ofthe specification are met or not met.When a statement of compliance with a specification is madeomitting the measurement results and associated uncertainties,the laboratory shall record those results and maintain them forpossible future reference.When statements of compliance are made, the uncertainty ofmeasurement shall be taken into account.

17025 requires that reor bad, or that a statemnon-compliance) to a mspecification be made.
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 29 

9. collective uncertainty

statement

5.10.4.1 In addition to the requirements listed in 5.10.2,calibration certificates shall include the following, wherenecessary for the interpretation of calibration results:b) the uncertainty of measurement and/or a statement of

compliance with an identified metrological specificationor clauses thereof;

17025 requires that an measurement result be where a statement of cuncertainty must be ac

10. traceability to national

standards statement

c) evidence that the measurements are traceable (see note 2

in 5.6.2.1.1).

17025 requires that thusually national standa

11. signature/title of laboratory

approval person

5.10.2 j) the name(s), function(s) and signature(s) or equivalent

identification of person(s) authorizing the test report orcalibration certificate;

17025 requires the sigassuming responsibilitresults. Laboratory acas the Approved Signa

12. calibration results data (asfound & as-left) 5.10.4.3 When an instrument for calibration has been adjusted or repaired, the calibration results before and after adjustment or

repair, if available, shall be reported.

17025 requires reportileft data only if the caadjusted/repaired but rmaintained of the data 

Additional reporting requirements of 17025 are as follows:5.10.2h) reference to the sampling plan and procedures used by

the laboratory or other bodies where these are relevantto the validity or application of the results;

k) where relevant, a statement to the effect that the results

relate only to the items tested or calibrated.

5.10.4.1a) the conditions (e.g., environmental) under which the

calibrations were made that have an influence on themeasurement results;

No equivalent NUPIC 
From the NUPIC Checklist Relevant requirement(s) of ISO/IEC 17025 Comments

NISTIR 6989 30 

5.10.4.4 A calibration certificate (or calibration label) shall not

contain any recommendation on the calibration interval exceptwhere this has been agreed with the client. This requirement maybe superseded by legal regulations.5.10.5 Opinions and interpretationsWhen opinions and interpretations are included, the laboratoryshall document the basis upon which the opinions andinterpretations have been made. Opinions and interpretationsshall be clearly marked as such in a test report.

Note: verify that quality programdocument specifies record retentionperiod; and determineimplementation effectiveness.

4.12.1.2 All records shall be legible and shall be stored and

retained in such a way that they are readily retrievable infacilities that provide a suitable environment to prevent damageor deterioration and to prevent loss. Retention times of recordsshall be established.

17025 requires that threcord retention requineeds of its customers regulatory requiremen
NISTIR 6989 31 May 2003

Appendix A

ISO/IEC 17025:1999 Table of Contents

Foreword

Introduction

1. Scope

2. Normative references

3. Terms and definitions

4. Management requirements

4.1 Organization

4.2 Quality system

4.3 Document control

4.4 Review of requests, tenders and

contracts

4.5 Subcontracting of tests and calibrations

4.6 Purchasing services and supplies

4.7 Service to the client

4.8 Complaints 

4.9 Control of nonconforming testing

and/or calibration work

4.10 Corrective action

4.11 Preventive action

4.12 Control of records

4.13 Internal audits 

4.14 Management reviews

5 Technical requirements 

5.1 General

5.2 Personnel

5.3 Accommodation and environmental

conditions

5.4 Test and calibration methods and

method validation

5.5 Equipment

5.6 Measurement traceability

5.7 Sampling

5.8 Handling of test and calibration items

5.9 Assuring the quality of test and

calibration results

5.10 Reporting the results

Annex A (informative) Nominal cross-references to ISO 9001:1994 and ISO 9002:1994

Annex B (informative) Guidelines for establishing applications for specific fields

Bibliography