![](media/image1.png){width="6.915277777777778in"
height="9.652083333333334in"}

**Preface**[]{#__RefHeading___Toc96940331 .anchor}

This publication was originally written by Ken Fraley, metrologist with
the State of Oklahoma, and Georgia Harris, physical scientist with the
NIST Weights and Measures Division. Ideas from final users regarding
publication content were sought at the 1994 National Conference on
Weights and Measures meeting held in San Diego, CA. The publication was
written to provide guidance for calibration laboratories in their desire
to provide improved precision mass calibrations and for the NIST Weights
and Measures Division to ensure uniform evaluation of laboratories
seeking to make mass measurements and be accredited at the advanced
level of mass calibration.

Since the first edition in 1995, numerous practical questions have been
raised and additional input has been sought from mass calibration
experts. This edition seeks to enhance the original publication and
provide additional guidance. Copies of Standard Operating Procedures 5
and 28 (3-1 Weighing Designs and Advanced Weighing Designs respectively)
are included as an appendix. Additional weighing designs, equations for
between-time standard deviations, and updates for uncertainty analysis
are included as well.

**About the Authors**

Ken Fraley is a metrologist with the State of Oklahoma. He has over 10
years of experience in making precision mass measurements using weighing
designs as described in this publication. He has carried out extensive
experimentation and implemented measurement assurance programs to ensure
that measurements made in the Oklahoma laboratory are consistent and
uniform with those at the national level. As background experience, Ken
developed the first draft of this document based on discussions with
NIST and early users of weighing designs in the State laboratories, and
provided a critical review of Advanced Laboratory Auditing Program (LAP)
problems submitted after the first Advanced Mass Metrology seminar in
1993. He has also coordinated and analyzed several interlaboratory
comparisons among laboratories working at the advanced level described
in this publication. He has analyzed data, prepared preliminary and
final analyses, and presented results of many interlaboratory
comparisons conducted at basic, intermediate and advanced levels. In
this edition, Ken has clarified ideas that now have had over 10 years of
refinement, has introduced extensive spreadsheet usage, and provided
additional graphic content.

Georgia Harris is the Group Leader of the Laboratory Metrology Group in
the NIST Weights and Measures Division. She provided direction and
encouragement to Ken in developing the ideas and provided editorial
support for the initial publication. In this revision, she has provided
updated copies of SOP 5, SOP 28, ideas and content for spreadsheet
analysis, and enhancements based on answering many technical questions
from laboratories working at this level, and from reviewing numerous
annual submissions for laboratories seeking formal Recognition at this
level. The Weights and Measures Division is responsible for providing
technical support and guidance to the State legal metrology laboratories
to ensure uniformity in the legal metrology measurement infrastructure;
this publication is intended to provide support and guidance not only
for State weights and measures laboratories, but also for other
calibration laboratories seeking to implement advanced weighing designs.

The authors wish to express their thanks to M. Carroll Croarkin (NIST)
for providing between-time standard deviation formulae and assistance
regarding updating mass calibration uncertainties to meet the ISO Guide
to the Expression of Uncertainty in Measurement, to Jerry L. Everhart
(JTI Systems, formerly with EG&G Mound) for providing guidance in
Process Measurement Assurance Programs, and to all of the metrologists
who regularly participate in WMD training and regional meetings for
their questions, comments, and desire to make precision mass
measurements to the best of their capabilities. For the 2005 update, the
authors greatly appreciate the efforts of Hung-kung Liu of the NIST
Statistical Engineering Division in supplying missing factors for the
between-time standard deviation equations and for his critical review of
the document.

**Contents**

**Preface** [iii](#__RefHeading___Toc96940331)

**Program Objective** [7](#__RefHeading___Toc96940332)

**Program Prerequisites** [8](#__RefHeading___Toc96940333)

Training [8](#__RefHeading___Toc96940334)

Facilities [8](#__RefHeading___Toc96940335)

Equipment [9](#__RefHeading___Toc96940336)

Standards [10](#__RefHeading___Toc96940337)

**Advanced LAP Problems** [11](#__RefHeading___Toc96940338)

LAP Problem 1 [12](#__RefHeading___Toc96940339)

LAP Problem 2 [12](#__RefHeading___Toc96940340)

LAP Problem 3 [12](#__RefHeading___Toc96940341)

Follow up [12](#__RefHeading___Toc96940342)

**Establishing Measurement Controls** [15](#__RefHeading___Toc96940343)

Process Evaluation [15](#__RefHeading___Toc96940344)

Data Input [17](#__RefHeading___Toc96940345)

Handling the Output [18](#__RefHeading___Toc96940346)

**Reviewing Mass Code Report** [19](#__RefHeading___Toc96940347)

**Graphs and Control Charts** [20](#__RefHeading___Toc96940348)

Critical Graphs [20](#__RefHeading___Toc96940349)

Optional Graphs [21](#__RefHeading___Toc96940350)

**Proficiency Tests** [21](#__RefHeading___Toc96940351)

**Evaluation Criteria for Proficiency Tests**
[22](#__RefHeading___Toc96940352)

Verification of Laboratory Values [23](#__RefHeading___Toc96940353)

Verification of the Laboratory Precision
[23](#__RefHeading___Toc96940354)

**File Management** [23](#__RefHeading___Toc96940355)

**Software Management** [24](#__RefHeading___Toc96940356)

Distribution [24](#__RefHeading___Toc96940357)

Licensing and Software Quality Assurance
[24](#__RefHeading___Toc96940358)

Updating [24](#__RefHeading___Toc96940359)

Approved Weighing Designs [24](#__RefHeading___Toc96940360)

**Documentation of Standard Operating Procedures**
[25](#__RefHeading___Toc96940361)

**Traceability and Calibration Intervals**
[25](#__RefHeading___Toc96940362)

**Formulae and Calculations** [26](#__RefHeading___Toc96940363)

**Advanced Mass Calibration and Measurement Assurance Program**

**for State Calibration Laboratories**

**Program Objective**[]{#__RefHeading___Toc96940332 .anchor}

This publication provides guidelines for evaluating data from advanced
mass calibrations and for establishing measurement assurance programs in
precision mass calibration laboratories. The NIST Weights and Measures
Division (WMD) will use these guidelines when evaluating advanced mass
calibration data for State laboratories that request technical support,
Recognition, and/or NVLAP accreditation.

Advanced mass calibrations use weighing designs, such as those found in
NBS Handbook 145[^1] (SOP 4, 5), NISTIR 6969, Selected Publications[^2],
NBS Technical Note 952[^3], and the NIST/SEMATECH e-Handbook of
Statistical Methods[^4] that require the use of computer software (mass
code) for the data reduction. These weighing designs are normally used
when high precision (low uncertainty) mass measurement results are
sought, although weighing designs can be used at any uncertainty level.
The uncertainty reported using advanced weighing designs is based on the
historically observed process of similar measurements and is very
dependent upon correct procedures for defining these processes.

NIST calibrations provide traceable standards at one point in time. The
major advantage in this program is the ability to evaluate
reference/working standards and the measurement process over time,
providing ongoing assurance regarding accuracy and traceability of the
mass standards for both the laboratory and its customers. Ongoing
evaluation of the measurement process provides the laboratory with data
that can be used to establish or adjust calibration intervals for
reference/working standards. The measurement assurance program is also
critical for defining and reporting realistic uncertainties.

**Program Prerequisites**[]{#__RefHeading___Toc96940333 .anchor}

The following items are listed as general guidelines for a laboratory
conducting an internal evaluation of its program for suitability in the
advanced mass calibration program for States. These guidelines have been
established based on good measurement practices, good laboratory
practices, and a similar fee-funded program (NIST Mass MAP) operated by
the NIST Mass and Force Group. Many specific technical recommendations
are taken from NIST Handbook 143, State Weights and Measures Laboratory
Program Handbook, G. Harris, Editor, March 2003, and NIST Handbook
150-2G: NVLAP Calibration Laboratories, Technical Guide for Mechanical
Measurements, C. Douglas Faison and Carroll S. Brickenkamp, Editors,
March 2004. Deviations from recommendations are occasionally made when
data is available to support it; however, judgments should be made
carefully when evaluating data, since some deviations from these
practices will inadvertently increase measurement uncertainties and may
contribute to measurement errors.

Training[]{#__RefHeading___Toc96940334 .anchor}

-   Satisfactory completion of an Intermediate Metrology Training course
    and Laboratory Auditing Program (LAP) problems within the last five
    years are expected before attendance at the Advanced Mass Training
    course which is traditionally taught in odd-numbered years.

-   Although required for NIST Recognition, attendance at the regional
    measurement assurance program meetings is not required to perform
    advanced mass calibrations. However, regular updates on precision
    mass procedures and issues are often provided at the regional
    meetings.

-   Satisfactory completion of the Advanced Mass Training course and
    Advanced LAP problems and successful application of these guidelines
    are expected before Recognition or accreditation at the Echelon I
    level as described in NIST Handbooks 143 and 150-2G.

Facilities[]{#__RefHeading___Toc96940335 .anchor}

-   Environment \-- The temperature for the laboratory where mass
    measurements are made should be selected at a point between 20 ^o^C
    and 23  ^o^C, with allowable variation of ± 1 ^o^C (e.g., 22 ^o^C
    ± 1 ^o^C), with a maximum change of 0.5 ^o^C per hour. Air flow must
    be low enough so that it does not interfere with balance or mass
    comparator operation. Humidity should be set between 40 % and 60 %
    relative humidity (e.g., 45 % ± 5 %). Environmental conditions must
    be monitored according to technical criteria and measurements should
    not be made when prescribed conditions are not maintained. Deviation
    from stated environmental parameters requires a thorough evaluation
    of the impact of the deviation on measurement errors and
    uncertainties.

-   Vibration \-- The laboratory location and design should be such as
    to avoid or minimize potential sources of vibration that will
    interfere with precision mass calibration.

-   Cleanliness \-- Good housekeeping practices and cleanliness
    specifications are found in NIST Handbook 145, 143 and NIST/NVLAP
    Handbook 150-2G. Contamination from dust, hair, paper,
    shipping/packaging/storage materials like felt and velvet, and other
    air contaminants has been found to be a critical concern.

Equipment[]{#__RefHeading___Toc96940336 .anchor}

-   Computer and printer \-- A computer of sufficient memory and
    processing ability is essential. In the laboratory, every effort
    must be made to minimize the impact of introducing temperature
    gradients in the measuring areas. Laptops or other wireless or panel
    monitors are preferred to large heat-producing systems. The printer
    should be capable of routinely producing the 40-page reports
    generated by the mass code, and it should also be capable of
    printing graphics. The printer should not be located in the
    precision mass laboratory.

-   Balances \-- A list of the laboratory\'s balances and process
    control chart data should be submitted to WMD for evaluation.
    Control data for each balance should consist of the following:

> \- Balance manufacturer, model number and serial number;
>
> \- Capacity and resolution; and
>
> \- Pooled standard deviations (accepted within-process standard
> deviations, s~w~, and accepted long term standard deviation of the
> check standard, s~t~), showing number of degrees of freedom, loads,
> and specific weighing designs.

Laboratory balances will be evaluated to determine their suitability for
this program. Minimum balance performance specifications recommended for
this program are as follows:

+----------------------------+-----------------------------------------+
| # Loads                    | \*Standard Deviation should be \<:      |
+----------------------------+-----------------------------------------+
| 20 kg                      | 1.5 mg                                  |
+----------------------------+-----------------------------------------+
| 10 kg                      | 0.20 mg                                 |
+----------------------------+-----------------------------------------+
| 1 kg                       | 0.050 mg                                |
+----------------------------+-----------------------------------------+
| 600/500/400/200 g          | 0.025 mg                                |
+----------------------------+-----------------------------------------+
| 60/50/40/20 g              | 0.0050 mg                               |
+----------------------------+-----------------------------------------+
| 6/5/4/2 g                  | 0.0025 mg                               |
+----------------------------+-----------------------------------------+
| 600/500/400/200 mg         | 0.0013 mg                               |
+----------------------------+-----------------------------------------+
| 60/50/40/20 mg             | 0.00050 mg                              |
+----------------------------+-----------------------------------------+
| 6/5/4/2 mg                 | 0.00030 mg                              |
+----------------------------+-----------------------------------------+

> \* The standard deviations in this table were developed by calculating
> 1/3 of OIML Class E~1~ tolerances, accounting for typical
> uncertainties associated with the standard and other factors, and
> developed by working backwards from a target expanded uncertainty.
> Many laboratories will not have balances or processes that can achieve
> these results on a routine basis.

-   Barometer \-- A barometer having documented accuracy of ± 65 Pa (0.5
    mm Hg) with evidence of traceable measurement results (typically
    from an accredited laboratory) must be available.

-   Thermometers \-- Thermometers to measure air temperature having
    accuracy of ± 0.1 ^o^C with evidence of traceable measurement
    results (typically from an accredited laboratory) are required.
    Temperature measurements made at this level are generally made
    within the balance chambers.

-   Hygrometer/Psychrometer \-- Percent relative humidity should be
    measured with an accuracy of ± 5 % and have evidence of traceable
    measurement results (typically from an accredited laboratory).

-   An environmental recording device is critical for monitoring
    laboratory conditions. Even though a number of environmental
    corrections are made in the mass measurement process, ensuring
    environmental stability during the 24-hour period preceding a
    calibration (particularly for temperature) is important to ensure
    proper thermal and environmental equilibration of mass standards.

Standards[]{#__RefHeading___Toc96940337 .anchor}

-   Reference (formerly called Primary) Standards \-- A minimum of two
    1‑kg reference standards (four 1‑kg reference standards are
    recommended) with NIST calibration and density determination are
    needed. Calibration values should be less than two years old.
    Standards should be calibrated at least every two years unless a
    measurement assurance program that monitors the reference kilogram
    standards is in place and demonstrates ongoing stability and
    validity of the mass values. If only two reference standards are
    available, and if they are used with equal frequency, the
    measurement assurance program will not be considered adequate
    without some type of verification using standards from outside the
    laboratory that have recent NIST-calibration. GMP 11, Good
    Measurement Practice for Assignment and Adjustment of Calibration
    Intervals for Laboratory Standards, and GMP 13, Good Measurement
    Practice for Ensuring Traceability (or other equivalent procedure),
    should be implemented in the laboratory to document the traceability
    hierarchy and calibration intervals that the laboratory will follow.
    (See discussions on Proficiency Tests and Graphs and Control
    Charts.)

-   Check standards \-- Check standards (sometimes called control
    standards) are not required to be calibrated by NIST. However,
    having the check standards calibrated by NIST or a competent
    external source provides an effective mechanism to identify and
    evaluate biases that may be occurring in the measurement processes
    that would otherwise go undetected; having an external calibration
    is essential.

> The check standards must be one‑piece design (to provide the necessary
> stability and act as surrogates to the reference standards) with known
> densities and have been assessed to comply with limits on magnetic
> susceptibility as required in ASTM and OIML standards, in the
> following decade denominations: 1 kg, 100 g, 10 g, 1 g, 100 mg, 10 mg,
> 1 mg. "ASTM Class 1, Type 1, Grade S" or "OIML Class E~2~" verbiage
> may be used if specifying weights for purchase. Additional check
> standards above 1 kilogram are needed to handle the entire range of
> calibration services, e.g., 10 kg.

Measurement Assurance (Control)

-   The laboratory needs to have a measurement assurance (control)
    system already in place before trying to perform advanced mass
    calibrations. Practical, hands-on experience in the laboratory is
    essential to making good mass measurements. A current measurement
    assurance system and data are essential for demonstrating
    measurement proficiency and/or justifying any deviations from these
    recommendations.

**Advanced LAP Problems**[]{#__RefHeading___Toc96940338 .anchor}

See Figure 1 for a graphic view of the components required in a complete
analysis of the Advanced LAP problems.

Laboratory Auditing Program (LAP) problems are used for establishing a
baseline for the initialization of the check standards, to provide
initial data to assess the between‑time component of the measurement
process, for providing validation on uncertainty statements, and for
evaluating the proficiency level at which the laboratory uses the mass
code. The data collected in the LAP problems is reduced by each
laboratory using the mass code. Each qualified metrologist must complete
training and the Advanced LAP problems and be able to reduce and analyze
their own data. Each laboratory is responsible for graphing and
analyzing data when determining the \"in control\" or \"out of control\"
condition of their standards (see sections on Establishing Measurement
Control and Graphs and Control Charts). Observed surveillance values
must be compared to the reported NIST values to determine the level of
control. A copy of all data, data files, reports, graphs, and final
analysis are to be sent to WMD for evaluation along with the most recent
NIST calibration report for the standards used. The final written
analysis, demonstrating a thorough understanding of the measurement
assurance system and uncertainty analysis at this level of work will be
considered as the most critical component of the completed Advanced LAP
problems.

NOTE: Standards should not be cleaned using solvents during the initial
data-collection period as these tests will provide data for determining
the total and between-time standard deviations for each series. Cleaning
standards changes their mass values and may invalidate the calibration.
Cleaning plans and procedures must be documented as a part of the
laboratory procedures. At least 7 to 10 days are required for
environmental equilibration on standards that have been cleaned with
solvents prior to recalibration.

LAP Problem 1[]{#__RefHeading___Toc96940339 .anchor}: Ten (10) complete
runs on reference/working State standards from 1 kg to 1 mg. (Initial
data from the first one or two runs may be submitted to be sure the
laboratory is on the right track.)

LAP Problem 2[]{#__RefHeading___Toc96940340 .anchor}: Two (2) complete
runs on standards from 30 kg to 2 kg.

The series selected when ascending from 1-kg may be the same as those
used when descending; however, a single restraint is usually used and
the between-time standard deviation formulae must be derived.

If the laboratory maintains both metric and avoirdupois standards with
NIST traceability, one additional LAP problem should be conducted. The
avoirdupois standards start at 1 pound and usually will require a "cross
over" from the kilogram. Special tare weights (92.815 g) can be obtained
to facilitate this process.

LAP Problem 3[]{#__RefHeading___Toc96940341 .anchor}: Two (2) complete
runs on standards from 50 lb to 1 μlb.

Follow up[]{#__RefHeading___Toc96940342 .anchor}: Note measurement
assurance guidelines and traceability for using the initial LAP problems
in continuing measurement assurance.

Note: Two complete runs is not adequate to provide data for establishing
initial limits nor for establishing a baseline for acceptable
measurement assurance nor for validating uncertainties at this level. If
the laboratory plans to provide internal calibration results or extended
service to customers, additional data must be obtained and analyzed.
Process statistics determined with limited data must have uncertainties
that reflect the actual degrees of freedom. The laboratory may limit
their application of this process to the smaller mass levels (e.g., 1 kg
to 1 mg, or 100 g to 1 mg). LAP problems 1 and 2 may be combined if a
limited range will be used for this level in the laboratory. For
example, ten runs from 10 kg to 1 mg, would be another acceptable
approach. If the laboratory does not plan to use the mass code for
avoirdupois standards, LAP problem 3 is not required.

![](media/image2.png){width="6.331944444444445in"
height="8.559722222222222in"}

![](media/image3.jpeg){width="6.719444444444444in"
height="8.529861111111112in"}

![](media/image4.wmf){width="6.539583333333334in" height="6.7625in"}

**Establishing Measurement Controls**[]{#__RefHeading___Toc96940343
.anchor}

Process Evaluation[]{#__RefHeading___Toc96940344 .anchor}

For each combination of a weighing design, specific load, and specific
balance (e.g., a 4‑1 design, at a 1‑kg load, on an AT 1005 balance), a
measurement control process should be defined and data collected to
characterize both the standard uncertainty of the process, s~w~, and the
standard uncertainty for the standard over time, s~t.~ These values are
essential for correctly reducing measurement data and calculating the
uncertainty assigned to each mass value in a report. This process is
more than simple statistical process control because the assigned values
for the standards and check standards are verified for accuracy with
each run.

In a surveillance test of reference/working standards from 1‑kg to 1‑mg,
there are seven series as characterized below:

+---------+----------------------+------------------+----------------+
| Series  | Weighing Design      | Decade Load      | Balance        |
|         |                      |                  |                |
|         | (Tech Note 952)      |                  | (examples      |
|         |                      |                  | only)          |
+=========+======================+==================+================+
| 1       | A.1.2 (1, 1, 1, 1)   | 1 kg             | AT 1005        |
|         | (4-1)                |                  |                |
+---------+----------------------+------------------+----------------+
| 1A      | A.1.4 (1, 1, 1,      | 1 kg             | AT 1005        |
| (       | 1, 1) (5-1)          |                  |                |
| option) |                      |                  |                |
+---------+----------------------+------------------+----------------+
| 2       | C.2 (5, 3, 2, 1, 1,  | 600, 500, 400,   | AT 1005        |
|         | 1)                   | 200 g            |                |
+---------+----------------------+------------------+----------------+
| (op     | A.1.2 or A.1.4       | 100 g            | AT 106, CC111  |
| tional) |                      |                  |                |
+---------+----------------------+------------------+----------------+
| 3       | C.2 (5, 3, 2, 1, 1,  | 60, 50, 40, 20 g | AT 106, CC111  |
|         | 1)                   |                  |                |
+---------+----------------------+------------------+----------------+
| 4       | C.2 (5, 3, 2, 1, 1,  | 6, 5, 4, 2 g     | UMT-6, CC6     |
|         | 1)                   |                  |                |
+---------+----------------------+------------------+----------------+
| 5       | C.2 (5, 3, 2, 1, 1,  | 600, 500, 400,   | UMT-6, CC6, S5 |
|         | 1)                   | 200 mg           |                |
+---------+----------------------+------------------+----------------+
| 6       | C.2 (5, 3, 2, 1, 1,  | 60, 50, 40, 20   | UMT-6, CC6, S5 |
|         | 1)                   | mg               |                |
+---------+----------------------+------------------+----------------+
| 7       | C.1 (5, 3, 2, 1, 1)  | 6, 5, 3, 2, 1 mg | UMT-6, CC6, S5 |
|         | or C.2               |                  |                |
+---------+----------------------+------------------+----------------+

NOTE: When data is reduced using the mass code, switching balances
within a series may result in artificially low process standard
deviations and will result in errors in between-time calculations.

As noted earlier, the Advanced LAP problems are the minimum
recommendations for collecting data that will begin to characterize the
measurement process. Ten complete runs on the reference/working
standards, 1‑kg through 1‑mg (all seven series reduced using the mass
code), must be made. An additional series at a 100‑g load is essential
to establish a secondary starting point for the calibration of 100‑g
kits. This means that initial data will be collected for at least these
eight series.

The section on Calculations in SOP No. 28 shows how to calculate each of
the standard uncertainty values that should be entered in the mass code
data file once data from the initial ten runs is available. Meanwhile,
because statistical data for the new process and check standards may be
unavailable, simulated values, based on knowledge of each measurement
process, should be used in order for the mass code to reduce data. For
the first ten runs, process standard deviation values, based on previous
three‑ones (also called three in one, abbreviated 3-1) weighing designs
or multiple double substitutions for each balance are satisfactory.
Using this process, only the size of the uncertainties and the
statistical tests will be affected when data is reduced; the mass values
are not affected. However, once actual data is available, the simulated
data in the data file must be replaced, and final reports generated. All
reports should be considered as "draft" during the process of gathering
the initial data; the statistics and uncertainty values must be updated
prior to performing the final data analysis.

Proper characterization of the measurement process is more critical when
using advanced weighing designs in decade series with the mass code than
when using routine mass comparisons with one‑to‑one standards. This is
primarily because the statistics used by the mass code distribute the
uncertainty from the starting restraint (reference standard)
proportionally among all the weights in each series. Also, the standard
uncertainty of the process (previously called random error) is
distributed among all the weights in each particular series. This type
of data reduction allows the mass code to assign smaller uncertainties;
however, the validity of these uncertainties is very dependent upon a
well-characterized measurement process. A major difference between
advanced weighing designs using the mass code, and routine calibrations
such as the 3‑1 weighing design and the double substitution, is that the
mass code uses the uncertainty of the starting restraint only. The 3‑1
design and the double substitution use the uncertainty of a single
standard at each denomination. In a 1‑kg to 1‑mg mass code calibration,
the starting restraint portion of the uncertainty is distributed among
all other denominations and soon becomes negligible around five grams.
Therefore, weights below five grams are primarily dependent upon the
standard uncertainty of the process when assigning an uncertainty to a
test weight.

Data Input[]{#__RefHeading___Toc96940345 .anchor}

The \[1993\] Fortran version of the mass code requires input for
statistical measurement control parameters in four locations:

> 1\) Line eight: \"ran err\" or \"random error\" for the standard is
> entered first; this hasn\'t been used and zero must be entered;
>
> 2\) Line eight: \"sys err\" or \"systematic error\" for the standard,
> s~r~, is entered second; this value is taken from the \[NIST\]
> calibration report and should be divided by the k‑factor that was used
> in reporting the uncertainty (see calculations in SOP No. 28 for
> formula to be used when using more than one restraint);
>
> 3\) Line thirteen: the within‑process standard deviation is the same
> as the standard uncertainty of the process, s~w~. This value is used
> for the F-test and is entered at the beginning of line thirteen on the
> first series (at the beginning of the line that contains the
> sensitivity weight data for subsequent series). The value is based on
> pooled data for observed standard deviations for the process (see
> calculations in SOP No. 28 for formula to be used); and
>
> 4\) Line thirteen: the between-time standard deviation, s~b~, is
> entered at the end of line thirteen on the first series (at the end of
> the line that contains sensitivity weight data for subsequent series)
> and is the calculated value that measures the variation of the value
> for the check standard over time, s~t~, less the contribution from the
> standard uncertainty of the process (see the calculations in SOP No.
> 28 for formula to be used.)

In Lab Wizard 1.0, the C++, Windows version of the mass code, these four
parameters are identified in the data entry area called \"Description of
Weights.\" They are identified in Restraint Specifications and
Statistical Parameters sections as: \"Random error,\" \"Systematic
error,\" \"Standard deviation,\" and \"Between std. dev.\". These items
correspond exactly to the Fortran version of the mass code and are
entered in the same location in the data file.

NOTE: Both the Fortran and Lab Wizard 1.0 versions of the mass
calibration software have been modified to conform to the NIST policy on
uncertainty (see NIST Technical Note 1297) as far as possible.
References to \"systematic error\" have been changed to \"type B
uncertainty\"; references to \"random error\" have been changed to
\"type A uncertainty\"; and references to \"uncertainty\" have been
changed to \"expanded uncertainty.\" The type B uncertainties are
calculated as one standard deviation estimates for systematic error, and
the type A uncertainties are calculated as one standard deviation
estimates for random error. The expanded uncertainties are calculated as
the root sum squares of the type A and type B uncertainties multiplied
by a coverage factor of two.

However, to preserve the integrity of the statistical control procedure
for mass calibrations, the operation of the code deviates from the NIST
policy in the evaluation of type A and type B uncertainties. The policy
defines type A (random) uncertainties in a global manner; i.e., as a
function of both local phenomena (balance precision and long-term
measurement precision) and hierarchical phenomena (uncertainties
associated with previously assigned values of reference standards). The
test for statistical control for each series requires a standard
deviation based only on local phenomena. The code does not distinguish
between these two requirements, and it will produce an improper t-test
if the hierarchical uncertainties are treated as random components. The
solution to this conflict is to distinguish between local and
hierarchical uncertainties and to define hierarchical uncertainties as
type B uncertainties.[^5]

Handling the Output[]{#__RefHeading___Toc96940346 .anchor}

To establish measurement controls once the mass code has been run,
certain data must be extracted from each series and placed in a
spreadsheet or database for storage, analysis, and measurement
assurance:

Test Number;

Operator ID;

Date of Test;

Balance ID;

Restraint ID;

Check Standard ID;

Check Standard Nominal Value;

Starting Restraint Number;

Calibration Design ID;

Average Corrected Temperature in Degrees C;

Average Corrected Pressure in Pascals;

Average Corrected Humidity in Percent;

Average Computed Air Density in mg/cm^3^;

Observed Standard Deviation of the Process;

Accepted Standard Deviation of the Process;

Degrees of Freedom;

F-Ratio;

Observed Correction of the Check Standard;

Accepted Mass Correction of the Check Standard;

t-Value; and

Expanded Uncertainty assigned to the Check Standard.

When either version of the mass code is run, it produces two files, one
called \"control\" and one called \"statis.\" The \"control\" file
contains a string of data for each series that contains: the month, day,
year, check standard identification, observed value for the check
standard, balance identification, process standard deviation, degrees of
freedom, weighing design identification, average temperature, range of
temperature during the measurement, average pressure, range of pressure
during the measurement, average humidity, range of humidity during the
measurement, air density, range of air density during the measurement,
operator identification, and a flag for process control results (0 = ok;
a number 1, 2, or 3 (depending on the version of the mass code used)
flags that the check standard failed, the observed standard deviation
failed, or both failed). This file can be imported to a spreadsheet
using a parse function so that each item is entered into a separate cell
and saved. The \"statis\" file contains F-test and t-test data for each
series run in the mass code. Unless an assignable blunder is detected in
the data, all out of control series must be saved in the control chart
file or the statistical limits will gradually become artificially small
resulting in an increased number of failed tests.

Each of these files should be saved by another name and/or in another
directory immediately after each run of the mass code because the data
is not cumulatively saved. These files contain information for the
preceding mass code reduction only, and are written over with each run
of the mass code.

**Reviewing Mass Code Report**[]{#__RefHeading___Toc96940347 .anchor}

Several sections within the mass code report should be reviewed for
adequacy. Key areas are observation values, the F-test, and the t-test.
If unusually high t-test or F-test values are observed, one should check
for data entry errors first.

Evaluate the F‑test to make sure that the observed standard deviation
agrees statistically with the accepted standard deviation of the
process. An F‑value is quoted and immediately below this value in the
report is a statement that shows whether the F‑test passed or failed.

Evaluate the t‑test to verify that the observed mass correction of the
check standard agrees statistically with the accepted mass correction of
the check standard. A t‑value is quoted and immediately below the
t-value in the report is a statement that says whether the check
standard is in‑control or out‑of‑control.

Should either test fail and no data-entry errors are found, the series
should be rerun. If the process is gradually changing, the t-values or
F-values will usually fall in a range from two to nine. If statistical
data is graphed properly, trends can be identified before they become
critical. If many tests fail, it could suggest a change in the
measurement process in which case the data is combined with the other
data to define the new measurement parameters. If failed data with no
attributable errors is routinely and incorrectly discarded, statistical
limits will be artificially reduced. Uncertainty values based on such
data will be invalid.

**Graphs and Control Charts**[]{#__RefHeading___Toc96940348 .anchor}

Critical Graphs[]{#__RefHeading___Toc96940349 .anchor}

Standard Control Charts -- When advanced mass calibrations are used for
surveillance testing, each weight (500 g to 1 mg) involved may be
graphed separately. These data and charts can be used to verify
calibration values and to determine appropriate calibration intervals.
When appropriate for reference/working standards, a new accepted value
may be calculated as either the mean of all values, or the predicted
value from the linear fit of the data at a time six months in the
future.

When the measurement process has been sufficiently characterized and
advanced mass calibrations are used for routine calibrations, a graph
must be prepared for the check standard at each decade to properly
characterize each measurement process. Analysis of the data provides the
statistics for calculating the between‑time standard deviations and can
be used to verify standard and check standard values. The standard
deviation over time is calculated from either the standard deviation
about the mean, or the residual standard deviation from the linear fit.
The standard deviation is later compared with the process deviation to
detect if there is a between-time component of error in the measurement
process. As noted for surveillance testing, the accepted values can be
calculated in one of two ways.

Measurements and control charts for two external or \"monitoring\" 1-kg
check standards are recommended to verify the calibration values for the
two 1-kg reference standards. Measurements are made between these two
external standards and the reference standards using a 4-1 weighing
design. The external kilogram standards may be part of a circulating
mass package with recent NIST calibration (as used in proficiency
testing) or may be maintained in the laboratory but must receive less
frequent (less than 25 % as often) use than the reference standards.
Analysis of calibration results over time as recorded on control charts
can provide a realistic estimate of calibration uncertainty and allow
the investigation of drift for standard values due to time and use.
Historical analysis can also help set calibration intervals.

Process charts \-- Control charts for the process standard deviation
(for each series/balance) are used to establish process variability and
an accepted within-process standard deviation, s~w~; each point can note
the degrees of freedom differentiating between series. Standard
deviations for each balance are plotted versus time. Plots are critiqued
for outliers and degradation. A new accepted value is calculated by
pooling the standard deviation for each balance.

Optional Graphs[]{#__RefHeading___Toc96940350 .anchor}

Summation Graphs -‑ With a measurement control program in place for 3-1
weighing designs that shows the values for summations of standards, data
from the mass code reports may be plotted with those values (when the
3‑1 design uses the summation as the check standard). A number of items
should be plotted versus time: the mean for the 3‑1 summation values,
the upper and lower control/warning limits, the mass code summation
values and uncertainties, and a line showing the NIST calibration value.
This graph may look cluttered, but it provides enormous insight to the
relationships between the NIST value, the mass code values, and the 3‑1
values for the same group of weights. It also provides a comparison
between the separate measurement processes. The 3‑1 values may show
greater precision than the mass code values. This is because in some
cases the 3‑1 values may be assigned using a balance with a lower
standard deviation of the process than the balance used for the mass
code.

F‑values -‑ Graphing the F‑values for a particular series can show
trends in the process and can evaluate the appropriateness of the
assigned process parameters. The F‑values are plotted chronologically
with the mean. The graph should be analyzed for trends and for uniform
distribution above and below the F-value of 1.00.

t‑values -‑ Graphing the t‑values of each series allows visualization of
the extent of agreement between the observed and accepted value of the
check standard. It also permits a comparison with the current
measurement process. The t‑values are plotted chronologically with the
mean. The graph should be analyzed for trends and for uniform
distribution above and below the t-value of 1.00.

**Proficiency Tests**[]{#__RefHeading___Toc96940351 .anchor}

Interlaboratory comparisons (proficiency tests) should be conducted at
least once every four years at the advanced level and may consist of two
kits with an entire set of standards: 1 kg through 1 mg. Each regional
measurement assurance group will have a schedule that addresses
compliance with the WMD PT policy. Charts should be prepared with an
\"uncertainty bar\" format with lines showing the lab average, lab
median, the NIST value and NIST upper and lower uncertainties. There
should be two charts for each denomination (one for each kit). A third
chart should be a scatter plot with a \"Youden\" analysis, with the
center of the circle at the two lab medians and the diameter of the
circle based upon the \"residual standard deviation\" of the
participating laboratories. This type of analysis will provide useful
information about potential errors in the laboratory, about the
uncertainty reported, and about drift of artifacts during the
intercomparisons. This information provides opportunity for evaluating a
laboratory\'s capability to meet stated uncertainties.

**Evaluation Criteria for Proficiency
Tests**[]{#__RefHeading___Toc96940352 .anchor}

Verification of the NIST Value

The NIST value will be verified by comparison with the median of all
accepted laboratory results. A standard deviation among laboratories and
an overall median are calculated using all participating lab values.
Using only those values that fall within the two standard deviation
limits, an adjusted standard deviation (a measure of reproducibility
between laboratories) and an adjusted round robin median are calculated.
These new statistical values are used in the evaluation. Verification of
the NIST value (VE) is based on the following formula:

The normalized error (E~normal~) concept (which is used internationally)
is used to verify the accepted value for each artifact by using the
Verification Error (VE). The normalized error is simply a ratio of the
difference between the observed and accepted values and the combined
uncertainty (at k=1) in the process combined by the root sum square
method. The VE value must be less than one to verify the NIST value.
When VE is equal to or greater than one, the standards should be
calibrated by NIST and the new NIST values and uncertainties should be
used to evaluate the proficiency test. The limits on the verification
test require the adjusted median value to agree with the reference value
within very tight limits. These tight limits are essential for
evaluation of proficiency when many laboratories are working at
state-of-the art levels.

Verification of Laboratory Values[]{#__RefHeading___Toc96940353 .anchor}

After verification of the NIST value, the following formula is used to
evaluate the acceptance of each laboratory value (NVLAP Handbook 150,
Procedures and General Requirements, July 2001.):

E~normal~ calculated for a laboratory must be less than one to pass this
test. The laboratory uncertainty and the reference value uncertainty are
calculated at a 95 % confidence level.

Verification of the Laboratory Precision []{#__RefHeading___Toc96940354
.anchor}

This criterion evaluates and validates the reported uncertainty of the
laboratory for its suitability to the level against which the laboratory
is being evaluated. At the highest level of mass calibration, the
uncertainty assigned by the participating laboratory should be less than
the tolerance. There are several perspectives regarding the use of
tolerance and uncertainty ratios; this is only a general guideline and
is not intended to be a requirement. However, if the laboratory will
determine compliance to OIML or ASTM standards, the expanded uncertainty
at k=2, must be less than 1/3 of the applicable tolerance.

$$U_{\text{lab}}\text{ < }\text{Tolerance}U_{\text{lab}}\text{Tolerance}$$

NOTE: Criteria for determining satisfactory/unsatisfactory compliance
for proficiency tests may be revised in the future to handle
uncertainties based on 95 percent confidence intervals rather than 99.7
percent confidence intervals.

**File Management**[]{#__RefHeading___Toc96940355 .anchor}

Mass Code reports that have been generated and printed do not need to be
saved on a hard disk; they can take up valuable space on the computer.
Instead, a directory may be created to store each data file. Data files
take up little disk space and the complete report can be generated as
needed.

Once the ten basic runs are made on the standards (LAP Problem 1) and
the process parameters are defined, the original data files can be
updated with the actual process parameters (replace simulated data with
observed data), and final reports can be generated. These new reports
will contain the same mass values, but the quoted uncertainties will
reflect the true process more closely.

**Software Management**[]{#__RefHeading___Toc96940356 .anchor}

Distribution[]{#__RefHeading___Toc96940357 .anchor}

Mass code software will be distributed by the NIST Weights and Measures
Division upon acceptable completion of the Advanced Mass Metrology
seminar. Lab Wizard software is "personalized" for each laboratory: the
name of the laboratory and of the metrologist are embedded in the
software and reports are generated containing the name of the laboratory
and the metrologist.

Licensing and Software Quality Assurance[]{#__RefHeading___Toc96940358
.anchor}

Each copy of mass code software distributed by WMD will be serialized,
and a list of trained metrologists and their laboratories is maintained.
All practical steps have been taken at NIST to ensure correct results
when the software is used with proper data files. However, each
laboratory must verify this independently and must document the
verification. Each licensee must agree to refrain from copying or
transferring software to others who have not participated in Advanced
Mass training unless WMD gives permission in writing. WMD will only
provide technical support to metrologists who have participated in the
Advanced Mass training from the NIST Weights and Measures Division.

Updating[]{#__RefHeading___Toc96940359 .anchor}

The mass code will be periodically updated and new versions will be
released to trained metrologists when available.

Approved Weighing Designs[]{#__RefHeading___Toc96940360 .anchor}

The Weights and Measures Division recognizes a variety of weighing
designs such as those found in NBS Technical Notes 844 and 952, and the
NIST/SEMATECH e-Handbook. However, weighing designs are used throughout
the world with variations from those presented in NIST publications.
Metrologists should use good judgment in developing unusual weighing
designs and may submit them to NIST for review or validation whenever
appropriate. Any designs submitted to NIST for review should be
accompanied by sufficient experimental data to provide adequate
evaluation. Metrologists should consider, and be able to justify,
variations in weighing designs, the selection and use of check
standards, length of designs and time restraints particularly with
respect to drift, selection of balance, use of sensitivity weights,
etc., to suit particular calibration applications. With the use of
electronic mass comparators, length of time during a design and fatigue
of the operator (which affect design selection) are of less concern than
with the older mechanical balances. When developing new designs, another
consideration should be that weights of equal nominal values should have
the same uncertainty. NIST/WMD strongly recommends designs that
incorporate check standards for process and standard verification.

Any unusual weighing designs not submitted to NIST for review will be
subject to critical review during on-site assessments.

**Documentation of Standard Operating
Procedures**[]{#__RefHeading___Toc96940361 .anchor}

To help with laboratory compliance to NVLAP Handbook 150, ANSI/NCSL
Z540-1-1994, and ISO/IEC 17025, SOP No. 28 "Recommended Standard
Operating Procedures for Using Advanced Weighing Designs" was developed
and is included with this update as an appendix.

**Traceability and Calibration Intervals**[]{#__RefHeading___Toc96940362
.anchor}

Ensuring traceability and providing documentation of how traceability is
maintained is a critical concern for customers and for accreditation
bodies when evaluating a laboratory. Measurement traceability for mass
measurements can be maintained through two reference kilograms that have
been calibrated at NIST as long as appropriate mass calibration and
measurement assurance procedures are used (and documented). The process
described in this document provides guidance for laboratories on how to
maintain adequate traceability and uncertainty needs. See GMP 11 and GMP
13 for additional examples of traceability hierarchies and calibration
interval requirements. Each laboratory must have an implementation
policy that covers the requirements in GMP 11 and GMP 13 to ensure
traceability and appropriate calibration intervals. Both of these GMPs
are posted on the NIST website.

The assigned LAP problems can be used to initiate an appropriate
measurement assurance program and prepare graphs. The data from the
analysis and preparation of the graphs must be evaluated against
original NIST values for the reference/working mass standards and check
standards as appropriate. Additional data collected periodically is
added to the original graphs. Data should be updated periodically and
evaluated. How often data is updated will depend on the laboratory
workload.

Each laboratory must document which standards are used at each level in
their traceability hierarchy process, what specific measurement
assurance is in place at each step, and how often intercomparisons are
conducted. The measurement assurance program, as described, is fully
integrated into the actual calibration process. Therefore, this is not
an exercise used to provide data for an accreditation body, but actually
provides checks on the system that the laboratory can use to ensure that
each measurement performed for a customer is accurate and traceable,
with validated uncertainties.

It is critical for laboratories to participate in interlaboratory
comparisons that provide periodic checks on the measurement process. The
data must be correlated with the measurement assurance program to be
meaningful. Any discrepancies indicate the need for further
investigation and possible need for calibration of the reference
standards. Laboratories must participate in this level of
intercomparison on a frequency no greater than four years (per WMD
policy published in NISTIR 7082, Proficiency Test Policy and Plan (for
State Weights & Measures Laboratories), G. Harris and J. Gust, January
2004.)

**Formulae and Calculations**[]{#__RefHeading___Toc96940363 .anchor}

The following items are calculated using formulae located in SOP No. 28
"Recommended Standard Operating Procedure for Using Advanced Weighing
Designs" which is included in this edition as an appendix.

-   s~r~ -- The standard uncertainty of the starting restraint in the
    first series.

-   s~w~ -- The within-process standard deviation.

-   s~b~ -- The between‑time standard deviation for each particular
    series .

-   Effective densities for summation standards.

-   Effective cubical coefficients of expansion for summation standards.

SOP 5 is inserted here as Appendix A for printing only; it is kept
separate from the document for website downloads.

SOP 28 is inserted here as Appendix B for printing only; it is kept
separate from the document for website downloads.

[^1]: Taylor, John K. And Henry V. Oppermann, NIST \[NBS\] Handbook 145,
    Handbook for the Quality Assurance of Metrological Measurements,
    November 1986.

[^2]: Selected Laboratory and Measurement Practices, and Procedures, to
    Support Basic Mass Calibrations

[^3]: Cameron, J. M., M. C. Croarkin, and R. C. Raybold, Technical Note
    952, Designs for the Calibration of Standards of Mass, June 1977.

[^4]: *NIST/SEMATECH e-Handbook of Statistical Methods*,
    http://www.itl.nist.gov/div898/handbook/, 2005.

[^5]:
    > Croarkin, M. C., Internal NIST Correspondence, July & August 1993.