February 2012 

NASA/TM-2012-217335/Volume II 

NESC-RP-08-00448 

 

Probabilistic Requirements (Partial) Verification 

Methods Best Practices Improvement 

Variables Acceptance Sampling Calculators: Empirical 

Testing 

Kenneth L. Johnson/NESC 

Langley Research Center, Hampton, Virginia 

K. Preston White, Jr. 

University of Virginia, Charlottesville, Virginia 
NASA STI Program . . . in Profile

 Since its founding, NASA has been dedicated to 

the advancement of aeronautics and space science. 

The NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

 The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. It 

collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI program 

provides access to the NASA Aeronautics and Space 

Database and its public interface, the NASA Technical 

Report Server, thus providing one of the largest 

collections of aeronautical and space science STI in 

the world. Results are published in both non-NASA 

channels and by NASA in the NASA STI Report 

Series, which includes the following report types: 

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

programs and include extensive data or 

theoretical analysis. Includes compilations of 

significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers, but having 

less stringent limitations on manuscript length 

and extent of graphic presentations. 

• TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis. 

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

• CONFERENCE PUBLICATION. Collected 

papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or co-sponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from NASA 

programs, projects, and missions, often 

concerned with subjects having substantial 

public interest. 

• TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and 

technical material pertinent to NASA’s mission. 

 Specialized services also include creating custom 

thesauri, building customized databases, and 

organizing and publishing research results. 

 For more information about the NASA STI 

program, see the following: 

• Access the NASA STI program home page at 

http://www.sti.nasa.gov

• E-mail your question via the Internet to 

help@sti.nasa.gov

• Fax your question to the NASA STI Help Desk 

at 443-757-5803 

• Phone the NASA STI Help Desk at 

443-757-5802 

• Write to: 

 NASA STI Help Desk 

 NASA Center for AeroSpace Information 

 7115 Standard Drive 

 Hanover, MD 21076-1320 
National Aeronautics and 

Space Administration 

Langley Research Center 

Hampton, Virginia 23681-2199 

 

February 2012 

NASA/TM-2012-217335/Volume II 

NESC-RP-08-00448 

 

Probabilistic Requirements (Partial) Verification 

Methods Best Practices Improvement 

Variables Acceptance Sampling Calculators: Empirical 

Testing 

Kenneth L. Johnson/NESC 

Langley Research Center, Hampton, Virginia 

K. Preston White, Jr. 

University of Virginia, Charlottesville, Virginia 
Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 

The use of trademarks or names of manufacturers in the report is for accurate reporting and does not 

constitute an official endorsement, either expressed or implied, of such products or manufacturers by the 

National Aeronautics and Space Administration. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

1 of 48

NESC Request No.: TI-08-00448

Volume II

Probabilistic Requirements (Partial) Verification Methods Best 

Practices Improvement

Variables Acceptance Sampling Calculators: 

Empirical Testing

October 27, 2011
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

2 of 48

NESC Request No.: TI-08-00448

Approval and Document Revision History 

NOTE: This document was approved at the October 27, 2011, NRB. This document was 

submitted to the NESC Director on January 30, 2012, for configuration control.

Approved: Original Signature on File 1/30/12

NESC Director Date

Version Description of Revision Office of Primary 

Responsibility

Effective Date

1.0 Initial Release Kenneth L. Johnson, Statistics 

and Trending, Marshall Space 

Flight Center

10/27/11
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

3 of 48

NESC Request No.: TI-08-00448

Table of Contents

Volume II: Technical Assessment Report

1.0 Notification and Authorization.........................................................................................7

2.0 Signature Page....................................................................................................................8

3.0 Team List............................................................................................................................9

4.0 Abstract.............................................................................................................................10

5.0 Introduction......................................................................................................................11

6.0 Test Protocol.....................................................................................................................11

7.0 Results...............................................................................................................................12

7.1 Attributes (Binomial) Plan.................................................................................... 12

7.2 Exponential Plan (Lower Limit)........................................................................... 13

7.3 Exponential Plan (Upper Limit)............................................................................ 16

7.4 Normal Plan (-known)........................................................................................ 20

7.5 Normal Plan (-unknown).................................................................................... 23

7.6 Gamma Plan (Lower Limit).................................................................................. 25

7.7 Gamma Plan (Upper Limit) .................................................................................. 27

7.8 Weibull Plan (Lower Limit).................................................................................. 29

7.9 Weibull Plan (Upper Limit).................................................................................. 32

7.10 Inverse Gaussian ................................................................................................... 35

7.11 Poisson (Upper Limit)........................................................................................... 41

8.0 Summary and Conclusions .............................................................................................43

9.0 Acronyms List ..................................................................................................................45

10.0 References.........................................................................................................................45

11.0 Appendices........................................................................................................................45

List of Figures

Figure 7.1-1. Attributes Simulation Results: Distribution Functions and Critical Values 

for X Distributed Binomial (777,p) Under (a) Null Hypothesis p0 = 0.005 and 

(b) Alternate Hypotheses and p1 = 0.001 for an Upper Limit on the Number of 

Nonconforming Observations ............................................................................. 12

Figure 7.2-1. Exponential Lower Limit Simulation Inputs: Distribution Functions and 

Critical Values for X with Lower Limit xmin = 1000 Under (a) the Null 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

4 of 48

NESC Request No.: TI-08-00448

Hypothesis p0 = 0.001 (0 = 999,500) and (b) Alternate Hypotheses and 

p1 = 0.005 ( 1 = 199,500)................................................................................... 13

Figure 7.2-2. Comparison of the Exponential (Lower Limit) and Attributes Sampling OC 

Curves for the Test Case ..................................................................................... 14

Figure 7.2-3. Exponential Lower-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. ............. 15

Figure 7.3-1. Exponential Upper Limit Simulation Inputs: Distribution Functions and 

Critical Values for X with Upper Limit xmax = 10,000 and Under (a) Null 

Hypothesis p0 = 0.001 (0 = 1447.65) (b) Alternate Hypotheses and 

p1 = 0.005 (1 = 1887.4)...................................................................................... 17

Figure 7.3-2. Comparison of the Exponential (upper limit) and Attributes Sampling OC 

Curves for the Test Case ..................................................................................... 18

Figure 7.3-3. Exponential Upper Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. ............. 19

Figure 7.4-1. Normal -known, Lower-Limit, Simulation Inputs: Distribution Functions 

and Critical Values for X with Lower Limit xmin = 1000 under (a) the Null 

Hypothesis p0 = 0.001 (0 = 1258) and (b) Alternate Hypotheses and 

p1 = 0.005 (1 = 1309)......................................................................................... 20

Figure 7.4-2. Comparison of the Normal Plan (-known) and Attributes Sampling OC 

Curves for the Test Case ..................................................................................... 21

Figure 7.4-3. Normal (-known), Lower-Limit, Simulation Results: Sampling Distributions 

for the Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. ............. 22

Figure 7.5-1. Normal (-Unknown), Lower-Limit, Simulation Inputs: Distribution 

Functions and Critical Values for X with Lower Limit xmin = 1000 under 

(a) the Null Hypothesis p0 = 0.001 (0 = 1258) and (b) Alternate Hypotheses 

and p1 = 0.005 (1 = 1309). Ovals are 99-percent Confidence Bounds on 

the Scatter............................................................................................................ 24

Figure 7.6-1. Gamma Lower-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Lower Limit xmin = 1000 under (a) the Null Hypothesis 

p0 = 0.001 (0 = 3377.79) and (b) Alternate Hypotheses and p1 = 0.005 

(1 = 3117.79) ..................................................................................................... 25

Figure 7.6-2. Gamma Lower-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A under the (a) Null and (b) Alternate Hypotheses; 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

5 of 48

NESC Request No.: TI-08-00448

Scatterplots of the Sample Statistics under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. ............. 26

Figure 7.7-1. Gamma Upper-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Upper Limit xmax = 10,000 Under (a) the Null Hypothesis 

p0 = 0.001 (0 = 4413.58) and (b) Alternate Hypotheses and p1 = 0.005 

(1 = 5587.57) ..................................................................................................... 27

Figure 7.7-2. Gamma Upper-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. ............. 29

Figure 7.8-1. Weibull Lower-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Lower Limit xmax = 10,000 Under (a) the Null Hypothesis 

p0 = 0.001 (0 = 1898.10) and (b) Alternate Hypotheses and p1 = 0.005 

(1 = 1712.00) ..................................................................................................... 30

Figure 7.8-2. Comparison of the Weibull Lower-limit Normal Plan and Attributes 

Sampling OC Curves for the Test Case............................................................... 31

Figure 7.8-3. Weibull Lower-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. ............. 32

Figure 7.9-1. Weibull Upper-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Upper Limit xmax = 10,000 Under (a) the Null Hypothesis 

p0 = 0.001 (0 = 7841.64) and (b) Alternate Hypotheses and p1 = 0.005 

(1 = 8121.07) ..................................................................................................... 33

Figure 7.9-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes 

Sampling OC Curves for the Test Case............................................................... 34

Figure 7.9-3. Weibull Upper-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; 

Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. ............. 35

Figure 7.10-1. IG Lower-Limit Simulation Inputs: Distribution Functions and Critical Values 

for X with Upper Limit xmin = 1000 Under (a) the Null Hypothesis p0 = 0.001 

(0 = 1502.00) and (b) Alternate Hypotheses and p1 = 0.005 (1 = 1413.00)..... 36

Figure 7.10-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes 

Sampling OC Curves for the Test Case............................................................... 37

Figure 7.10-3. IG Lower-Limit Simulation Results: Sampling Distributions for the Allowable 

Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the 

Sample Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 

99-percent Confidence Bounds on the Scatter. ................................................... 38
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

6 of 48

NESC Request No.: TI-08-00448

Figure 7.10-4. IG Lower-Limit Simulation Results Applying a Normal Sampling Plan: 

Sampling Distributions for the Allowable Limit A Under the (a) Null and 

(b) Alternate Hypotheses; Scatterplots of the Sample Statistics Under the 

(c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter.......................................................................................... 40

Figure 7.10-5. Comparison of the IG and Normal Distributions with the Same Mean and 

Standard Deviation .............................................................................................. 41

Figure 7.11-1. Poisson Upper-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Upper Limit xmax = 10 Under (a) the Null Hypothesis 

p0 = 0.001 and (b) Alternate Hypotheses and p1 = 0.005 .................................... 42

Figure 7.11-2. Poisson Simulation Results: Distribution Functions and Critical Values for X

Distributed Poisson (27,p) with Limit xmax = 10 and Under (a) Null Hypothesis 

p0 = 0.001 and (b) Alternate Hypotheses and p1 = 0.005 .................................... 43

List of Tables

Table 8.0-1. Summary of Test Results for Continuous Variables Given Lower Limit 

xmin = 1000 and Nominal OC (p0,) = (0.001,0.2) and (p1,) = (0.001,0.1)......... 44

Table 8.0-2. Summary of Results for Continuous Variables Given Upper Limit 

xmax = 10,000 and Nominal OC (p0,) = (0.001,0.2) and (p1,) = (0.001,0.1) ..... 44

Table 8.0-3. Summary of Results for Discrete Variables.......................................................... 44
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

7 of 48

NESC Request No.: TI-08-00448

Volume II: Technical Assessment Report

1.0 Notification and Authorization

The NASA Engineering and Safety Center (NESC) was requested to improve on the Best 

Practices document produced for NESC assessment 07-001-E: Verification of Probabilistic 

Requirements for the Constellation Program (CxP) by giving a recommended procedure for 

using acceptance sampling by variables techniques as an alternative to the potentially resourceintensive acceptance sampling by attributes method given in the document.

An NESC out-of-board activity was approved on January 9, 2008. Mr. Kenneth Johnson at 

Marshall Space Flight Center (MSFC) was selected to lead this assessment. The assessment plan 

was approved at the NESC Review Board (NRB) on March 20, 2008. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

8 of 48

NESC Request No.: TI-08-00448

2.0 Signature Page 

Submitted by: 

Team Signature Page on File – 2-7-12

Kenneth L. Johnson Date

Significant Contributors:

K. Preston White, Jr. Date

Signatories declare the findings and observations compiled in the report are factually based from 

data extracted from Program/Project documents, contractor reports, and open literature, and/or 

generated from independently conducted tests, analyses, and inspections. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

9 of 48

NESC Request No.: TI-08-00448

3.0 Team List

Name Discipline Organization/Location

Core Team

Kenneth Johnson Statistics and Trending MSFC

K. Preston White, Jr. Department of Systems and Information 

Engineering University of Virginia

Chris Johansen MTSO LaRC

Administrative Support

Christina Williams Technical Writer LaRC/ATK
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

10 of 48

NESC Request No.: TI-08-00448

4.0 Abstract

Acceptance sampling is a method for verifying lot quality or performance requirements using 

sample data. In this paper, the results of empirical tests intended to assess the accuracy of 

acceptance sampling plan calculators implemented for six variable distributions—binomial, 

exponential, normal, gamma, Weibull, inverse Gaussian (IG), and Poisson are presented. In 

general, these results support the accepted wisdom that variables acceptance plans are superior to 

attributes (binomial) acceptance plans, in the sense that these provide comparable protection 

against both producer’s and consumer’s risk at reduced sampling cost. For the Gaussian and 

Weibull plans, however, there are ranges of the shape parameters for which the required sample 

sizes are in fact larger than the corresponding attributes plans, dramatically so for instances of 

large skew. Tests also confirm that the published IG plan is flawed (White and Johnson, 2011). 

Appendix A provides the protocol for selecting a sampling plan calculator. Appendix B provides 

a note on hypothesizing an output distribution. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

11 of 48

NESC Request No.: TI-08-00448

5.0 Introduction

Acceptance sampling by attributes (ASA) assesses the quality of a lot based on the number of 

nonconforming items discovered in a random sample drawn for inspection. Inspection requires 

only a pass/fail determination for each item. Because it is conceptually straightforward, easily to 

implement, and can be applied to qualitative as well as quantitative performance measures, ASA 

is the first choice for sampling inspection. 

For quantitative performance measures, however, a pass/fail determination typically is 

accomplished by comparing the measured value to a limiting value, without regard to magnitude 

of conformance or nonconformance for each item tested. It seems reasonable that this additional 

information might be exploited to decrease the number of items that need to be inspected. This 

is the rationale behind acceptance sampling by variables (ASV). If there is adequate information 

to posit a distribution for the measure, then in many instances the ASV alternative translates into 

significantly smaller samples to achieve the same operating characteristic. While far more 

restrictive in its assumptions, ASV should be considered only when larger samples required by 

ASA is an issue. 

The objective of this paper is to provide an independent assessment of the accuracy of variables 

plans reported in the literature. Note that, with the exception of normal plans, the search for offthe-shelf ASV plan calculators was essentially fruitless. This scarcity strongly suggests that nonnormal ASV largely has been limited to an academic audience and not fully vetted in practice. 

The need to implement and test plans reported in the literature is especially important for those 

plans based on approximations. This paper discusses the results of empirical tests conducted 

using spreadsheet implementation of calculators developed for this purpose.

6.0 Test Protocol

The plans reviewed by White and Johnson (2011) were implemented as spreadsheet calculators 

and tested empirically using Monte Carlo simulation. The test protocol enforced a limit standard 

with (1) a specification limit on the measured variable X, either xmin or xmax, as the performance 

indicator I, (2) minimum reliability = 0.005, and (3) maximum consumer’s risk  = 0.100. 

Additionally, maximum producer’s risk of  = 0.200 was enforced. These particular test 

conditions were chosen as representative of certain high-level requirements in the design of 

spacecraft (White, et al., 2009). 

For each test, values were specified for the limit and for any distribution parameters assumed to 

be known or estimated. The null and alternative means 0 and 1 were determined such that, for 

a lower limit, F(xmin;0) = 0.001 and F(xmin;1) = 0.005. For an upper limit, 1  F(xmax;0) =

0.999 and 1  F(xmax;1) = 0.995. The corresponding (n,) sampling plan was then determined 

from the appropriate calculator. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

12 of 48

NESC Request No.: TI-08-00448

One-hundred thousand Monte Carlo trials were run for both the null and alternative distributions, 

each run comprising n observations as determined by the sampling plan. The proportions 



ˆ

and



ˆ

were estimated from the sampling distribution of the acceptance limit A(n,k). These estimates 

were compared to the specified operating characteristic to assess the accuracy of the plan. The 

efficiency of the variable sampling plan was determined by comparing the required sample size 

to that for the closest attributes sampling plan. 

7.0 Results

7.1 Attributes (Binomial) Plan

For the test operating characteristic (OC), the closest attributes sampling plan from the binomial 

calculator is (n,c) = (777,1), with null and alternative test points (p0,1  ) = (0.001,0.8280) and 

(p1,) = (0.005,0.0998). As shown in Figure 7.1-1, the requirement for integer sample size n

implies that the specified limits on the risks are enforced, but not necessarily as equalities. This 

plan is otherwise exact, in the sense that (1) the pass/fail determination is necessarily distributed 

binomial (n,p) and requires no assumptions regarding the distribution of X and (2) there is a 

single unknown parameter p and additional parameters need not be estimated approximately. 

(a) (b)

Figure 7.1-1. Attributes Simulation Results: Distribution Functions and Critical Values for X 

Distributed Binomial (777,p) Under (a) Null Hypothesis p0 = 0.005 and (b) Alternate 

Hypotheses and p1 = 0.001 for an Upper Limit on the Number of Nonconforming 

Observations
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

13 of 48

NESC Request No.: TI-08-00448

7.2 Exponential Plan (Lower Limit)

Consider a lower limit of xmin = 1000 for an exponentially distributed random variable X with 

unknown mean . For the test OC, the associated the null and alternative means are 

0 = 999,500 and 1 = 199,500, respectively, as shown in Figure 7.2-1. The variables plan from 

the exponential calculator is (n,k) = (2, 2.42722  103). 

(a) (b)

Figure 7.2-1. Exponential Lower Limit Simulation Inputs: Distribution Functions and 

Critical Values for X with Lower Limit xmin = 1000 Under (a) the Null Hypothesis p0 = 0.001

(0 = 999,500) and (b) Alternate Hypotheses and p1 = 0.005 ( 1 = 199,500)

Figure 7.2-2 compares the OC curves for the exponential and attribute plans. These curves are 

similar, with the exponential sampling plan modestly more conservative with respect to risks. 

Note that here and throughout, the OC curves are theoretical and derived from the plans, and 

may or may not agree with empirical results.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

14 of 48

NESC Request No.: TI-08-00448

Figure 7.2-2. Comparison of the Exponential (Lower Limit) and Attributes Sampling OC 

Curves for the Test Case

Simulations were run to estimate the distribution of the test statistic, the allowable limit 

ˆ

A k  

ˆ , 

where the estimated mean 



ˆ

is the sample mean 

X n( )

and n an k are given by the sampling 

plan. Figure 7.2-3(a) and (b) shows the sample distributions of 

A

ˆ

for the null and alternate 

hypotheses, respectively. From these distributions, it was observed that there was no practical 

difference between the estimated risks of 



ˆ = 0.200 and 



ˆ = 0.082 and those predicted by the 

OC curve.

Figure 7.2-3(c) and (d) are scatterplots of the sample mean and standard deviation for each of the 

100,000 simulation trials for the null and alternate hypotheses, respectively. Also shown on each 

scatterplot is the threshold value of A, the constant  = xmin/k = 411,993. This line divides each 

sample into two sets, with the conforming observations above the line and nonconforming 

below.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

15 of 48

NESC Request No.: TI-08-00448

(a) (b)

(c) (d) 

Figure 7.2-3. Exponential Lower-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample

Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter.

Each figure shows that the proportion of nonconforming observations subtended by the line is 

approximately 20 percent for the null distribution and the proportion of conforming observations 

subtended by the line is approximately 8.20 percent for the alternate distribution. Increasing the 

limiting value xmin will decrease the proportion conforming (increasing  and decreasing ). 

Increasing the value of k will have a similar effect.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

16 of 48

NESC Request No.: TI-08-00448

The respective 99-percent confidence bounds for each data set are shown as ovals on the 

scatterplots. The area subtended by these bounds depends on the number runs n. Increasing the 

number of runs will decrease the radius of the confidence bound, decreasing  and for fixed. 

The lower-limit exponential plan clearly outperforms the attributes plan, by providing equivalent 

risk protection at a dramatically reduced sampling cost. The number of simulations indicated by 

the binomial is n = 777, whereas the variables plans require only n = 2, which is 0.26 percent of 

the computational effort of the attributes plan. From Figure 7.2-1, it should be clear that this 

economy was achieved because the null and alternate distributions are distinct—observations 

that are typical of one distribution are rare for the other. The economy was further enhanced by 

the need to estimate only a single parameter sample from the data. 

It seems obvious, however, that two observations were inadequate to determine the exponential 

distributional form if it is otherwise unknown. Thus, in the case of exponentially distributed 

measurement, it appears that the number of trials required in many cases will be dictated by the 

need to determine the form of the output distribution, rather than the need to satisfy a limit 

standard.

7.3 Exponential Plan (Upper Limit)

Consider an upper limit of xmax = 10,000 for an exponentially distributed random variable X with 

unknown mean . For the test OC, the associated the null and alternative means are

0 = 1447.65 and 1 = 1887.4, respectively, as shown in Figure 7.3-1. The variables plan from 

the exponential calculator is (n,k) = (66,6.26922).
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

17 of 48

NESC Request No.: TI-08-00448

(a) (b)

Figure 7.3-1. Exponential Upper Limit Simulation Inputs: Distribution Functions and 

Critical Values for X with Upper Limit xmax = 10,000 and Under (a) Null Hypothesis 

p0 = 0.001 (0 = 1447.65) (b) Alternate Hypotheses and p1 = 0.005 (1 = 1887.4)

Figure 7.3-2 compares the OC curves for the exponential upper limit and attributes plans. These 

curves again are similar and almost identical at the OC points tested. Simulations were run to 

generate the distribution of the test statistic 

A k  

ˆ

for the exponential tests. Results are shown 

in Figure 7.3-3. There is no practical difference between the estimated risks of 



ˆ = 0.200 and 



ˆ = 0.082 and those shown on the OC curve.

The exponential upper-limit plan also outperforms the attributes plan, by providing equivalent or 

superior risk protection at a substantially reduced sampling cost. The number of simulations 

indicated by the binomial is n = 777, whereas the variables plans require n = 66, which is 

8.49 percent of the computational effort of the attributes plan. From Figures 7.3-1 and 7.3-3, it is 

clear that the null and alternate distributions are not nearly as distinct as in the case of a lower 

bound, owning to the skew. Thus, the economy is not as great as that for a lower bound, but is 

still impressive. Again, it may be that 66 observations are inadequate to determine the 

exponential distributional form and that the number of trials required in many cases will be 

dictated by this need, rather than the need to satisfy a limit standard.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

18 of 48

NESC Request No.: TI-08-00448

Figure 7.3-2. Comparison of the Exponential (upper limit) and Attributes Sampling OC 

Curves for the Test Case

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.01

Pa(p)

p

Expo Upper Limit Attributes
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

19 of 48

NESC Request No.: TI-08-00448

(a) (b)

(c) (d)

Figure 7.3-3. Exponential Upper Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample

Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

20 of 48

NESC Request No.: TI-08-00448

7.4 Normal Plan (-known)

Consider a lower limit of xmin = 1000 for a normally distributed random variable X with unknown 

mean  and known standard deviation  = 100. For the test OC, the associated the null and 

alternative means are 0 = 1309 and 1 = 1258, respectively, as shown in Figure 7.4-1. The 

variables plan from the norm calculator is (n,k) = (18,2.88632) is also shown.

(a) (b)

Figure 7.4-1. Normal -known, Lower-Limit, Simulation Inputs: Distribution Functions and 

Critical Values for X with Lower Limit xmin = 1000 under (a) the Null Hypothesis p0 = 0.001 

(0 = 1258) and (b) Alternate Hypotheses and p1 = 0.005 (1 = 1309)

Figure 7.4-2 compares the OC curves for the normal and attributes plans. These curves again are 

similar, with the normal plan modestly more conservative in the region tested. Simulations were 

run to generate the distribution of the test statistic 

0 A k   

ˆ 100

for the normal tests. Results 

are shown in Figure 7.4-3. There was no practical difference between estimated risks of 



ˆ = 0.193 and 



ˆ

= 0.096 and those predicted by the OC curve.

The normal plan outperformed the attributes plan, by providing equivalent or superior risk 

protection at a dramatically reduced sampling cost. The number of simulations indicated by the 

binomial was n = 777, whereas the variables plans require only n = 18, which is 2.32 percent of 

the computational effort of the attributes plan. Given the symmetry of the normal distribution, 

the same results for an upper bound were expected.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

21 of 48

NESC Request No.: TI-08-00448

Figure 7.4-2. Comparison of the Normal Plan (-known) and Attributes Sampling OC Curves 

for the Test Case
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

22 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.4-3. Normal (-known), Lower-Limit, Simulation Results: Sampling Distributions 

for the Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the 

Sample Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent 

Confidence Bounds on the Scatter. 

(a) (b)

(a) (b)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

23 of 48

NESC Request No.: TI-08-00448

7.5 Normal Plan (-unknown)

Test conditions are identical to the -known case, except that standard deviation is now



ˆ , and 

estimated from the data for each trial. The variables plan from the normal calculator is 

(n,k) = (88,2.88632) with OC curve identical to that for the -known case. 

Simulations were run to generate the distribution of the test statistic 

A k   

ˆ 100

for the normal 

tests. Results are shown in Figure 7.5-1. The estimated risks of 



ˆ = 0.191 and 



ˆ

= 0.109 are not 

quite those and predicted by the OC curve, with modestly lower producer’s risk and modestly 

higher consumer’s risk. The normal plan outperforms the attributes plan, by providing 

equivalent or protection at a dramatically reduced sampling cost. The number of simulations 

indicated by the binomial was n = 777, whereas the variables plans require only n = 88, which is 

11.3 percent of the computational effort of the attributes plan. Given the symmetry of the normal 

distribution, the same results for an upper bound was expected.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

24 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.5-1. Normal (-Unknown), Lower-Limit, Simulation Inputs: Distribution Functions 

and Critical Values for X with Lower Limit xmin = 1000 under (a) the Null Hypothesis 

p0 = 0.001 (0 = 1258) and (b) Alternate Hypotheses and p1 = 0.005 (1 = 1309). Ovals are 

99-percent Confidence Bounds on the Scatter 

(a) (b)

(c) (d)

(a) (b)

(c) (d)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

25 of 48

NESC Request No.: TI-08-00448

7.6 Gamma Plan (Lower Limit)

Consider a lower limit of xmin = 1000 for a random variable X distributed gamma with unknown 

shift parameter  and estimated shape and scale parameters, 



ˆ

10

and

ˆ

  337.779

. For the test 

OC, the associated null and alternative means were 0 = 3377.79 and 1 = 3140.79, as shown in 

Figure 7.6-1. The variables plan from the gamma calculator was (n,k) = (206,2.13126).

Simulations were run to generate the distribution of the test statistic 

0 A k   

ˆ 100

. Results are 

shown in Figure 7.6-2. There was no practical difference between estimated risks of 



ˆ

= 0.191 and 

ˆ

  0.096

and those prescribed. The gamma plan outperforms the attributes 

plan, by providing equivalent or superior risk protection at a reduced sampling cost. The number 

of simulations indicated by the binomial was n = 777, whereas the variables plans require 

n = 206, which is 25.5 percent of the computational effort of the attributes plan. 

Figure 7.6-1. Gamma Lower-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Lower Limit xmin = 1000 under (a) the Null Hypothesis p0 = 0.001 

(0 = 3377.79) and (b) Alternate Hypotheses and p1 = 0.005 (1 = 3117.79)

(a) (b)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

26 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.6-2. Gamma Lower-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample 

Statistics under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

27 of 48

NESC Request No.: TI-08-00448

Note, however, for the test OC and 



ˆ

3.431

, the variables plan required samples larger than the 

attributes plan. For example, 



ˆ

0.200

sampling plan from the gamma calculator was 

(n,k) = (1960,1.46100)—nearly three times larger than the attributes plan. Given the other 

advantages of attributes sampling, gamma plans would never be used under this condition. This 

contradicts the accepted wisdom that ASV will always be more efficient than ASA. The 

operative point was that the sample size for any variables plan should be compared to the 

corresponding attributes plan and the benefit of the potential computational savings achieved 

(if any) assessed in the light of the restrictive assumptions imposed by variables sampling.

7.7 Gamma Plan (Upper Limit)

Consider an upper limit of xmax = 10,000 for a random variable X distributed gamma with 

unknown shift parameter  and estimated shape and scale parameters, 



ˆ

10

and

ˆ

  441.358

. For the test OC, the associated the null and alternative means are 0 = 4413.58 and 

1 = 5587.57, respectively, as shown in Figure 7.7-1. The variables plan from the gamma 

calculator is (n,k) = (77,3.66693). 

(a) (b)

Figure 7.7-1. Gamma Upper-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Upper Limit xmax = 10,000 Under (a) the Null Hypothesis p0 = 0.001 

(0 = 4413.58) and (b) Alternate Hypotheses and p1 = 0.005 (1 = 5587.57)

Simulations were run to generate the distribution of the test statistic 

0 A k   

ˆ 100

for the 

Gamma test. Results are shown in Figure 7.7-2. The estimated risks of 



ˆ = 0.186 and 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

28 of 48

NESC Request No.: TI-08-00448

ˆ

  0.104

are not quite those and predicted by the OC curve, with modestly lower producer’s 

risk and modestly higher consumer’s risk.

The gamma plan outperforms the attributes plan, by providing equivalent or superior risk 

protection at a reduced sampling cost. The number of simulations indicated by the binomial is 

n = 777, whereas the variables plans require n = 77, which is 9.91 percent of the computational 

effort of the attributes plan. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

29 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.7-2. Gamma Upper-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample

Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 

7.8 Weibull Plan (Lower Limit)

Consider a lower limit of xmin = 1000 for a random variable X distributed Weibull with unknown 

shift parameter  and estimated shape and scale parameters, 



ˆ

10

and

ˆ

 1995.16

. For the test 

(a) (b)

(c) (d)

(a) (b)

(c) (d)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

30 of 48

NESC Request No.: TI-08-00448

OC, the associated the null and alternative means are 0 = 1898.10 and 1 = 1712.00, 

respectively, as shown in Figure 7.8-1. The variables plan from the Weibull calculator is 

(n,k) = (91,3.62929. Figure 7.8-2 compares the OC curves for the normal and attributes plans. 

These curves again are similar. 

(a) (b)

Figure 7.8-1. Weibull Lower-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Lower Limit xmax = 10,000 Under (a) the Null Hypothesis p0 = 0.001 

(0 = 1898.10) and (b) Alternate Hypotheses and p1 = 0.005 (1 = 1712.00)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

31 of 48

NESC Request No.: TI-08-00448

Figure 7.8-2. Comparison of the Weibull Lower-limit Normal Plan and Attributes Sampling 

OC Curves for the Test Case

Simulations were run to generate the distribution of the test statistic 

0 A k    

ˆ ˆ

for the Weibull 

test. Results are shown in Figure 7.8-3. The estimated risks of 



ˆ = 0.1889 and 

ˆ

  0.0794

were modestly more conservative than those predicted by the OC curve and there is 

no practical difference between the estimated risks and those predicted by the OC curve. The 

Weibull plan outperforms the attributes plan, by providing equivalent or superior risk protection 

at a reduced sampling cost. The number of simulations indicated by the Binomial n = 777, 

whereas the variables plans require n = 91, which is 11.7 percent of the computational effort of 

the attributes plan. 

Note that for the test OC and



ˆ

 2.157, the variables plan requires samples larger than the 

attributes plan and consequently would never be used. For example, 



ˆ

 2

and all else being 

equal, the sampling plan from the Weibull calculator is (n,k) = (1071,1.81126). This contradicts 

the accepted wisdom that ASV will always be more efficient than ASA.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

32 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.8-3. Weibull Lower-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample

Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 

7.9 Weibull Plan (Upper Limit)

Consider an upper limit of xmax = 10,000 for a random variable X distributed gamma with 

unknown shift parameter  and estimated shape and scale parameters, 



ˆ

10

and

ˆ

  3800.00 . 

For the test OC, the associated the null and alternative means are 0 = 7841.64 and 1 = 8121.07, 

(a) (b)

(c) (d)

(a) (b)

(c) (d)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

33 of 48

NESC Request No.: TI-08-00448

respectively, as shown in Figure 7.9-1. The variables plan from the gamma calculator was 

(n,k) = (156,2.17779). 

(a) (b)

Figure 7.9-1. Weibull Upper-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Upper Limit xmax = 10,000 Under (a) the Null Hypothesis p0 = 0.001 

(0 = 7841.64) and (b) Alternate Hypotheses and p1 = 0.005 (1 = 8121.07)

Figure 7.9-2 compares the OC curves for the normal and attributes plans. These curves again are 

very similar, with the normal plan modestly more conservative in the region tested. Simulations 

were run to generate the distribution of the test statistic 

0 A k   

ˆ 100

for the Weibull test. 

Results are shown in Figure 7.9-3. The estimated risks of 



ˆ = 0.1883 and 

ˆ

  0.0806

were 

modestly more conservative than those predicted by the OC curve.

The Weibull plan outperformed the attributes plan, by providing equivalent or superior risk 

protection at a reduced sampling cost. The number of simulations indicated by the Binomial 

n = 777, whereas the variables plans require n = 156, which is 20.1 percent of the computational 

effort of the attributes plan. Note that for the test OC and



ˆ

 0.4069, the variables plan requires 

samples larger than the attributes plan and consequently would never be used. For example, 



ˆ

0.200

sampling plan from the gamma calculator is (n,k) = (4368,9.43402)—more than five

times larger than the attributes plan. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

34 of 48

NESC Request No.: TI-08-00448

Figure 7.9-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes Sampling 

OC Curves for the Test Case

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.01

Pa(p)

p

Wiebull Upper Limit Attributes
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

35 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.9-3. Weibull Upper-Limit Simulation Results: Sampling Distributions for the 

Allowable Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample

Statistics Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence 

Bounds on the Scatter. 

7.10 Inverse Gaussian

Consider a lower limit of xmax = 1000 for a random variable X distributed Poisson with unknown 

shift parameter  and estimated mean 



ˆ = 1502 and shape parameter 

ˆ

 = 100,000. For the test 

(c) (d)

(c) (d)

(c) (d)

(c) (d)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

36 of 48

NESC Request No.: TI-08-00448

OC, the associated null and alternative means were 0 = 1502.00 and 1 = 1413.00, respectively, 

as shown in Figure 7.10-1. The variables plan from the IG calculator was (n,k) = (18,2.88632). 

(a) (b)

Figure 7.10-1. IG Lower-Limit Simulation Inputs: Distribution Functions and Critical Values 

for X with Upper Limit xmin = 1000 Under (a) the Null Hypothesis p0 = 0.001 (0 = 1502.00) 

and (b) Alternate Hypotheses and p1 = 0.005 (1 = 1413.00)

Figure 7.10-2 compares the OC curves for the normal and attributes plans. These curves were

similar, with the normal plan modestly more conservative in the region tested. Note, however, 

that the theoretical OC curve for the published plan did not match the empirical data, as shown 

below.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

37 of 48

NESC Request No.: TI-08-00448

Figure 7.10-2. Comparison of the Weibull Upper-Limit Normal Plan and Attributes Sampling 

OC Curves for the Test Case

The test statistic for the IG plan (Aminzadeh, 1996) is:

 min min min   

ˆ ˆ

A k x Q k x x          ˆ ˆ 1 1 ( , , .

Simulations were run to generate the distribution of the statistic and the results are shown in 

Figure 7.10-3. The estimated producer’s risk of 



ˆ = 0.173 was modestly conservative. 

However, the estimated consumer’s risk of

ˆ

  0.382

was unacceptably large—almost four 

times that specified—and far from that predicted for the theoretical OC. This confirms the error 

in derivation reported by White and Johnson (2011). The scatter diagrams plot the value of the 

sample mean 



ˆ

against the parameter group Q for all 100,000 simulation trials. It was apparent 

that the 99-percent confidence bounds on the simulation data are too large and that the 

(n,k) = (18,2.88632) sampling plan is too small. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

38 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.10-3. IG Lower-Limit Simulation Results: Sampling Distributions for the Allowable 

Limit A Under the (a) Null and (b) Alternate Hypotheses; Scatterplots of the Sample Statistics 

Under the (c) Null and (d) Alternate Hypotheses. Ovals are 99-percent Confidence Bounds on 

the Scatter. 

As a check, second simulation experiment was run using the same IG distribution, but this time 

following the normal sampling plan (n,k) = (88,2.88632) and corresponding normal test statistic. 

Results are shown in Figure 7.10-4. The larger sample size has the desired effect of compacting 

the confidence bounds. However, the estimated producer’s risk of 



ˆ = 0.743 was exceedingly 

(a)

(b)

(c) (d)

(a)

(b)

(c) (d)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

39 of 48

NESC Request No.: TI-08-00448

large—almost four times that specified— while the estimated consumer’s risk of 



ˆ = 0.020 was 

highly conservative. The reason for this is clear from Figure 7.10-5, which compares the IG and 

normal density functions with the same mean and standard deviation. Even though the skew in 

the IG distribution is modest, the lower tail of the normal distribution is considerably fatter than 

that of the IG. Therefore, the normal plan is conservative with respect to consumer’s risk for IG 

data unless these data are suitably transformed. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

40 of 48

NESC Request No.: TI-08-00448

 (a) (b)

 (c) (d)

Figure 7.10-4. IG Lower-Limit Simulation Results Applying a Normal Sampling Plan: 

Sampling Distributions for the Allowable Limit A Under the (a) Null and (b) Alternate 

Hypotheses; Scatterplots of the Sample Statistics Under the (c) Null and (d) Alternate 

Hypotheses. Ovals are 99-percent Confidence Bounds on the Scatter. 

(a) (b)

(c) (d)

(a) (b)

(c) (d)
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

41 of 48

NESC Request No.: TI-08-00448

Figure 7.10-5. Comparison of the IG and Normal Distributions with the Same Mean and 

Standard Deviation

Figure 7.10-3 shows the critical value of the test statistic for the normal plan, k = 2.88632, as a 

solid line. The effect of reducing this value to k = 2.60000 as a dashed line is also shown. The 

(n,k) = (88,2.60000), while approximate, was much closer to the desired plan for the test OC than 

either the normal or published IG plans.

7.11 Poisson (Upper Limit)

Consider an upper limit of xmax = 10 for a random variable X distributed Poisson with unknown 

mean . For the test OC, the associated null and alternative means were 0 =2.96051 and 

1 =3.79612, respectively, as shown in Figure 7.11-1. The variables plan from the gamma 

calculator was (n,c) = (27,88). 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

42 of 48

NESC Request No.: TI-08-00448

(a) (b)

Figure 7.11-1. Poisson Upper-Limit Simulation Inputs: Distribution Functions and Critical 

Values for X with Upper Limit xmax = 10 Under (a) the Null Hypothesis p0 = 0.001 and 

(b) Alternate Hypotheses and p1 = 0.005

Simulations were run to generate the distribution of the test statistic

0

n

j

j

Y X





. Results are 

shown in Figure 7.11-2. There was no practical difference between the estimated risks of 

estimated risks of 



ˆ = 0.192 and 



ˆ = 0.097 and those specified by the OC. 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

43 of 48

NESC Request No.: TI-08-00448

(a) (b)

Figure 7.11-2. Poisson Simulation Results: Distribution Functions and Critical Values for X 

Distributed Poisson (27,p) with Limit xmax = 10 and Under (a) Null Hypothesis p0 = 0.001 and 

(b) Alternate Hypotheses and p1 = 0.005

The Poisson plan dramatically outperformed the attributes plan, by providing equivalent or 

superior risk protection at a reduced sampling cost. The number of simulations indicated by the 

binomial was n = 27, whereas the variables plans require n = 777, which is 0.347 percent of the 

computational effort of the attributes plan. 

8.0 Summary and Conclusions

This paper presents the results of empirical tests designed to provide an independent assessment 

of the validity and accuracy of six published ASV procedures. These results are summarized in 

Tables 8.0-1 through 8.0-3. Overall, the plans were shown to provide adequate or superior 

protection against producer’s and consumer’s risks for samples substantially smaller than those 

required for the corresponding ASA plans. The exception is the IG plan, which was previously 

shown to be in error.
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

44 of 48

NESC Request No.: TI-08-00448

Table 8.0-1. Summary of Test Results for Continuous Variables Given Lower Limit xmin = 1000

and Nominal OC (p0,) = (0.001,0.2) and (p1,) = (0.001,0.1)

Variable n k



ˆ 

ˆ

nv/na

Exponential() 2

2.43 × 

103

0.200 0.082 0.003

Normal(, = 100) 18 2.886 0.191 0.097 0.023

Normal(,

ˆ

) 88 2.886 0.191 0.097 0.099

Gamma(



ˆ

10,

ˆ

  338,) 206 2.131 0.193 0.096 0.224

Weibull(



ˆ

10,

ˆ

 1995 ,) 91 3.623 0.189 0.079 0.117

IG(



ˆ = 1502, 

ˆ

 = 100,000,) 18 2.886 0.173 0.382 unusable

Table 8.0-2. Summary of Results for Continuous Variables Given Upper Limit xmax = 10,000 and 

Nominal OC (p0,) = (0.001,0.2) and (p1,) = (0.001,0.1)

Variable n k



ˆ 

ˆ

nv/na

Exponential() 66 6.26922 0.200 0.082 0.085

Gamma(



ˆ

10,

ˆ

  441,) 77 3.667 0.189 0.104 0.099

Weibull(



ˆ

10,

ˆ

  3800 ,) 156 3.623 0.188 0.081 0.201

Table 8.0-3. Summary of Results for Discrete Variables

Variable n c



ˆ

b nv/na

Binomial (n,p) 777 1 0.188 0.100 1

Poisson (n,p) 21 88 0.191 0.097 0.035

While this overall conclusion supports the assertion in the literature that ASV plans require 

smaller samples than ASA plans, it was also discovered that this assertion does not hold 

absolute. In particular, for gamma and Weibull variables with small shape parameters, the ASV 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

45 of 48

NESC Request No.: TI-08-00448

plans were in fact larger than the corresponding ASV plans. It is believed that this discovery is 

original and needs to be assimilated into the literature on acceptance sampling.

9.0 Acronyms List

ASA Acceptance Sampling by Attributes

ASV Acceptance Sampling by Variables

ATK Alliant Techsystems, Inc.

CxP Constellation Program

IG Inverse Gaussian

LaRC Langley Research Center

MSFC Marshall Space Flight Center

MTSO Management Technical Support Office

NESC NASA Engineering and Safety Center

NRB NESC Review Board

OC Operating Characteristic

10.0 References

1. Aminzadeh, M. S. (1996), "Inverse-Gaussian Acceptance Sampling Plans by Variables," 

Communications is Statistics--Theory and Methods 25(5)923-935.

2. White, K. P., Jr., Johnson, K. L., and Creasey, R. R., Jr. (2009). “Attribute acceptance 

sampling as a tool for verifying requirements using Monte Carlo simulation, Quality 

Engineering, 21(2)203-214). 

3. White, K. P., Jr., and Johnson, K. L. (2011), “Variables Acceptance Sampling Calculators: 

Derivations and Verification of Plans,” NASA Engineering Statistics Team, working paper 

available from the authors.

11.0 Appendices

Appendix A. Protocol for Selecting a Sampling Plan Calculator

Appendix B. Note on Hypothesizing an Output Distribution
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

46 of 48

NESC Request No.: TI-08-00448

Appendix A. Protocol for Selecting a Sampling Plan Calculator

START Determine the (n,c) attributes acceptance plan for your requirment Do you have the computing budget to run this plan? FINISH Use the (n,c) attributes plan

Draw the largest sample permitted by your computing budget Fit all supported distributions to the sample data and determine goodness of fit Do any of the distributions provide an accepatble fit?

FINISH There is no acceptance sampling plan for your budget. Consider revising budget and/or requirement.

Does more than one distribution provide an acceptable fit? FINISH Use the corresponding (n,k) plan with the lowest acceptance probability FINISH Use the corresponding (n,k) sampling plan

YES NO

YES

NO

NO YES

Hypothesize the form of the output distribution and determine the corresponding (n,k) variables acceptance plan for your requirement* Do you have the computing budget to run this plan? YES NO Runs the simulation and generate a sample of size n as given by the variables sampling plan

*See note

Do you have the computing budget to make additional runs? YES
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

47 of 48

NESC Request No.: TI-08-00448

Appendix B. Note on Hypothesizing an Output Distribution

The sampling plan calculators can be used to determine the number of simulation replications or 

runs needed, i.e., the number of Monte Carlo trials yielding the sample size n specified by the 

plan for a given requirement. However, the choice of which calculator to use depends on 

knowing the form of the distribution of the verification output. The size of sampling plans for 

different distributions can vary by orders of magnitude depending on this distribution.

Pass/fail output always have a binomial distribution and therefore the n can be determined using 

the binomial (attributes) calculator before any simulation runs are made. For variables output, 

the choice may not be as obvious.

If historical or test data exist for variables outputs that are thought to be similar in nature to the 

expected verification output, a verifier may be able to make use of such prior knowledge to guess 

what form the distribution might assume. S/he needn’t be concerned about the values of the 

distribution parameters at this step, just with the general shape of the distribution. The choice of 

distribution (hypothesis) and parameter estimates can be checked once data are generated for the 

verification analysis. 

There also are theoretical considerations that might suggest one or more distributions, or at least 

rule out some of the distributions available. For example, the Normal distribution often is a 

reasonable model for errors of various types, where the magnitude of the error is equally likely to 

be positive or negative and smaller errors are more common than larger errors. By virtue of the 

central limit theorem, the Normal distribution also is often a reasonable model for the sum of a 

large number of other random quantities, no matter how these quantities are distributed. Because 

the Normal distribution has a negative tail, however, it may not be a reasonable model for a time, 

or a count, or some other output that must be strictly positive. 

Because of the “memoryless property”, the exponential distribution is a suitable model for the 

time between the occurrence of events, if these occurrences have a constant rate and the numbers 

of occurrence in disjoint time periods are independent (i.e., the time of occurrence of one event 

does not influence the time of occurrence of subsequent events). Events might be the failure of a 

component or system, or the arrival of a customer needing service, or the placing of orders 

against an inventory. By the same property, the number of events occurring during any fixed 

period of time under these same assumptions is an integer count distributed Poisson.

The gamma and Weibull distributions frequently are used to model the time to some event, where 

the event might be a failure or the completion of some task, such as a service or repair. The 

Weibull is quite flexible in application, can have positive or negative skew, and frequently is 

used as the assumed distribution when modeling in the absence of data. 

The lognormal distribution (the normal distribution applied to the natural logarithm of the data) 

has applications similar to the gamma and Weibull. It takes on shapes similar to the gamma and 
NASA Engineering and Safety Center 

Technical Assessment Report

Document #:

NESC-RP08-00448

Version:

1.0

Title:

Probabilistic Requirements Verification Methods Best 

Practices Improvement

Page #:

48 of 48

NESC Request No.: TI-08-00448

Weibull with positive skew, but can have a large “spike” at the mode close to zero. There are 

other transformations to the Normal that may also be appropriate.

Before constructing a hypothesis and collecting any data, it is a good idea to consult a statistician 

to help in this activity. It is also important to be aware that the data, once generated, may result 

in a finding that the hypothesized distribution is not the conservative choice. The analyst, 

managers and other verifiers should have a contingency plan covering the possibility that the 

simulation trials already run are not sufficient for verification at the confidence level required.
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Memorandum

 4. TITLE AND SUBTITLE

Probabilistic Requirements (Partial) Verification Methods Best Practices 

Improvement

Variables Acceptance Sampling Calculators: Empirical Testing

5a. CONTRACT NUMBER

 6. AUTHOR(S)

Johnson, Kenneth L.; White, K. Preston, Jr.

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

L-20120 NESC-RP-08-00448

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 65 - Statistics and Probability

Availability: NASA CASI (443) 757-5802

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

The NASA Engineering and Safety Center was requested to improve on the Best Practices document produced for the NESC 

assessment, Verification of Probabilistic Requirements for the Constellation Program, by giving a recommended procedure for 

using acceptance sampling by variables techniques as an alternative to the potentially resource-intensive acceptance sampling 

by attributes method given in the document. In this paper, the results of empirical tests intended to assess the accuracy of 

acceptance sampling plan calculators implemented for six variable distributions are presented.

15. SUBJECT TERMS

acceptance sampling by attributes; acceptance sampling by variables; NASA Engineering and Safety Center; Constellation 

Program

18. NUMBER

 OF 

 PAGES

53

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

March 2008 - October 2011

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

869021.03.07.01.18

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA/TM-2012-217335/Volume II

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 02 - 2012