# Analysis of Variance for Item Differences in Verification Data with Unknown Groups

Authors: T. Burr; K. Martin; C. Norman; K. Zhao
Journal: Science and Technology of Nuclear Installations
 (2019)

---

## Abstract

For sequentially collected data, this paper introduces a lag-one differencing method to estimate the random error standard deviationδR and then uses the estimate δ^R to calculate a change detection threshold in a moving window method to detect shifts in the short-term systematic error. Performance results on simulated and real data are presented. Fortunately, the impact of having to perform change detection on the estimated short-term systematic and random error variances is anticipated to be modest or small. The motivating example arises from facilities under nuclear safeguards agreements, where inspector data collected during International Atomic Energy Agency (IAEA) verifications are compared to corresponding operator data. The differences between the operator and inspector values are evaluated using an application of analysis of variance (ANOVA). Typically, it is assumed that short-term systematic errors change across inspection periods, so inspection periods form the groups used in the ANOVA. In some data sets, it appears that the short-term errors have changed at other times, so change detection methods could be used to detect the actual change times.


---

## Body

## 1. Introduction and Background

For facilities under nuclear safeguards agreements, inspector data collected during International Atomic Energy Agency (IAEA) verifications are compared to corresponding operator data. The differences between the operator and inspector values are evaluated using an application of analysis of variance (ANOVA). It is typically assumed that short-term systematic errors change across inspection periods (groups); however, in some data sets, it appears that the short-term errors have changed at other times, so change detection methods could be used to detect the actual change times. This paper introduces a lag-one differencing method to estimate the within-group random error standard deviationδR and then uses the estimate δ^R to calculate a change detection threshold in a moving window method to detect shifts in the short-term systematic error. Performance results on simulated and real data are presented.An effective measurement error model for inspector measurement data must account for variation within and between groups, where in this context a group is usually (but not always) an inspection period. A model for multiplicative errors for the inspector (I) (and similarly for the operator O) is(1)Ijk=μjk1+SIj+RIjk,where for item k (from 1 to n) from group j (from 1 to g), Ijk is the inspector’s measured value, μjk is the true unknown value, SIj~N(0,δSI2) is the short-term systematic error in group j, and RIjk~N(0,δRI2) is a random error. The notation RIjk~N(0,δRI2) means that RIjk has a normal distribution with mean 0 and variance δRI2, and similarly for SIj~N(0,δSI2) [1]. For simplicity, it is assumed here that there are n measurements in each inspection period; allowing for a differing number of measurements in each inspection period is straightforward.The measurement error model in (1) sets the stage for applying ANOVA with random effects [1–4]. Neither RIjk nor SIj are observable. However, for various types of observed data, the variances δRI2 and δSI2 can be estimated. The variance of Ijk is σI2=σμ2+(μ-2+σμ2)(δSI2+δRI2), or for a fixed value of μjk, σI2=μjk2(δSI2+δRI2), where the item variance σμ2 is the variance of the μjk, defined as σμ2=∑j=1g∑k=1n(μjk-μ-)2/ng, and μ-={∑j=1g∑k=1nμjk}/ng is the overall mean of all the true values. The values d=(O-I)/O, by linear error variance propagation, have approximate variance δSO2+δRO2+δSI2+δRI2=δd2 (an accurate approximation for large δd, up to approximately δd= 0.60). Regarding notation, δR2=δRO2+δRI2 and δS2=δSO2+δSI2. This paper focuses on estimating δR and δS.


## 2. ANOVA

The usual ANOVA decomposition (which is a standard analysis technique) is(2)∑j=1g∑k=1ndjk-d-∙∙2=∑j=1g∑k=1ndjk-d-i∙2+n∑j=1gd-j∙-d-∙∙2=SSW+SSB,where djk=(ojk-ijk)/ojk, d-j∙=(1/n)∑k=1ndjk, and d-∙∙=(1/ng)∑j=1g∑k=1ndjk In (2), SSW is the within-group sum of squares, and SSB is the between-group sum of squares. In random effects ANOVA (the group means SIj~N(0,δSI2) are random), the expected value E(∑j=1g∑k=1n(djk-d-k∙)2)/(ng-1)=δR2 and E((n/g-1)∑j=1g(d-j∙-d-∙∙)2)=nδS2+δR2, so reasonable estimators are δ^R2=SSW/g(n-1) and δ^S2=(SSB/(g-1)-SSW/(g(n-1)))/n. It follows that E∑j=1g∑k=1n(djk-d-∙∙)2/(ng-1)=δR2+δS2(ng-n)/(ng-1). The vector (d--,SSW,SSB) is a complete sufficient statistic for (μ-,δR2,δS2) so by the Lehmann-Sheffé theorem, the estimators δ^R2 and δ^S2 have the minimum variance among all possible unbiased estimators (MVUE) [1]. However, biased estimators can have smaller mean-squared error (MSE), where MSEδ^R2=E{(δ^R2-δR2)2}andMSEδ^S2=E{(δ^S2-δS2)2}. Also, the MVUE property relies on the normality assumption. Therefore, other estimators are sometimes considered [1]; for example, setting δ^S2 to 0 if it is negative leads to a positive bias, but also to a lower MSE.Figure1(a) plots 10 simulated d=(O-I)/O values in each of 3 groups. The alarm limit on the right side of the plot in Figure 1(a) is estimated on the basis of the g = 3 sets of n = 10 values of d. The groups in data such as that in Figure 1(a) are typically inspection periods; however, in some data sets, it appears that the short-term errors have changed at other times, so change detection methods can be used to detect the actual change times. This paper introduces a lag-one differencing method to estimate the random error standard deviation δR and then uses the estimate δ^R to calculate a change detection threshold in a moving window method to detect shifts in the short-term error S. Performance results on simulated and real data are presented.Figure 1
Example (simulated) of 10d=(O-I)/O values in each of 3 groups. δRO=0.005,δSO=0.001,δRI=0.02,δSI=0.01,δT=0.05,μT=100.
(a)
( O - I ) / O (b)
O (c)
I (d)
Inspector versus operator data in group 1Data such as that in Figure1 are used for verification, and also for two main metrology purposes. One metrology purpose is to estimate performance values for the four RSDs, δSI2,δRI2,δSO2,andδRO2. For verification, statistical tests are applied to d=(O-I)/O, so a second metrology purpose is to estimate δR2 and δS2, by applying random effects one-way ANOVA to real paired relative difference data that are typically assumed to follow (2). Reference [5] evaluates impacts on alarm probabilities of using estimates of δR2 and δS2 (this paper’s focus) instead of the true quantities.Figure2 is four pairs of real operator and inspector data for each of three measurement periods (“groups”), plotted as (O-I)/O versus the measurement order. In the context of change detection for paired (O,I) data such as that in Figure 1 (simulated data) or Figure 2 (real data), there are situations where the default choice to group by inspection period seems to be contradicted by the data. The main goals for such paired (O,I) data are to(1)
estimate the within-group and between-group variance forO-I data (this paper’s focus),(2)
estimate thresholds that correspond to small false alarm probabilities for futureO-I data as in Figure 1,(3)
estimate the within-group and between-group variance for bothO and I data for international target values [4] and to use for error variance propagation in material balance evaluation.Figure 2
Example of real (a) (O-I)/O and (b)O for data from 3 groups, 4 measurements per group.
(a)
( O - I ) / O data (b)
O dataThis application is an example of random effects ANOVA, which is a standard analysis technique, but with the nonstandard feature that the group memberships are unknown. Some research issues arise in applying ANOVA to paired(O,I) data, including the following three:( 1 ) One-way random effects ANOVA that arises in the verification context sometimes with unknown groups has not been investigated in published literature; see Section 3.( 2 ) Although ANOVA dates to the 1930s [1], research continues regarding, for example, constructing intervals for δS2. Assuming that the R and S values are normally distributed, an exact CI can be constructed for δR2 using the χdf=g(n-1)2 distribution, but there are only approximate methods to construct CIs for δS2, because the distribution of δ^S2 is a difference of two independent χ2 random variables. Kraemer [6] proposes a modified CI construction method for δS2 and investigates impacts of nonnormality. Kraemer’s modified CI method is based on the chi-squared based Satterthwaite approximation method, [νδ^S2/χν,α/22,νδ^S2/χν,1-α/22], where the chi-squared degrees of freedom ν=2(δS2/var(δ^S2))2 for a 1-α CI, and also on the approximation var(δ^S2)≈(2/(g-1))(δS2+δR2/n)2+(2/gn-1)(δR2/n)2+(γS^-3)(δS^4/g) [7], where γδS^ is the kurtosis of the distribution of δ^S. If the R and S distributions are normal, then γδS^=0. This approach works reasonably well in simulations reported in [6]; however, it ignores the fact (unpublished) that cov(MSB,MSW)=(1/ng)(μ4-3δR4), where μ4 is the fourth moment of the distribution of S+R.( 3 ) A variance approximation issue arises in analysing (O-I)/O instead of (O-I)/μ (not available in practice).Linear first-term error variance propagation is accurate for quite large δd up to approximately 0.60. The results reported in Section 4 use drel=(O-I)/O to estimate δR and δS. Also, when checking whether future drel=(O-I)/O values are within alarm limits, there is no need to separately estimate δSO2andδRO2orδSI2andδRI2, so usual ANOVA is adequate, rather than the Grubbs’ ANOVA[2–5].


## 3. One-Way Random Effects ANOVA with Unknown Groups

Regarding research issue one, if wrong groupings are chosen, thenδ^R2 and δ^S2 can be adversely impacted. For example, Figure 3 plots an exaggerated example for ease of illustration, in which successive groups are incorrectly combined. To show that E(δ^R2) and E(δ^S2) are impacted by mismatch between the true and assumed grouping, consider the ANOVA decomposition equation (3), where djk=(ojk-ijk)/ojk for multiplicative models. In Figure 3, the groups 1 and 2, 3 and 4, and 5 and 6 are merged. Specifically, let N=ng=36 and the assumed values are na=12 and ga =3, but the true values are nt= 6 and gt= 6. One must take care to interpret terms because there are two values of n (nt= 6 and na=12) and two values of g (gt= 6 and ga = 3), depending on whether the true or merged groups are being used in the calculation.Figure 3
Illustration of incorrectly merging successive groups.Because the new group 1 has two different values ofS (one for the first 6 d values and another for the second 6 d values), the within-group (former groups 1 and 2) MS (which should estimate random error variance only) has expected value E(∑j=12∑k=16(djk-d-j∙)2)/(12-1)=(12-2)δR2/(12-1)+6(2-1)δS2/(12-1)+(2-1)δR2/(12-1)=δR2+6δS2/11>δR2.Therefore,E(δ^R2)=δR2+6δS2/11>δR2.The between-groups sum of squares also has the wrong expected value due to the fact that var(S1+S2) is being estimated, because the average of 2 successiveS values is used in each of the 3 (merged) groups. So, MSB=(n/g-1)∑j=1g(d-j∙-d-∙∙)2 has expected value 12×(δS2/2+δR2/12)=12δS2+δR2. Therefore, the estimate δ^S2= (MSB-MSW)/n has expected valueE(δ^S2)=(12(δS2/2+δR2/12)-(δR2+(6/11)δS2))/12=5δS2/11<δS2.The general expressions forE(δ^R2) and E(δ^S2) when merging successive pairs of groups of size n are(3)Eδ^R2=δR2+ntδS22nt-1,(4)Eδ^S2=nt-1δS22nt-1.Results (3) and (4) have been confirmed by simulation in the R statistical programming language [8] for data simulated from (1) such as that in Figure 3 (see Table 1). Notice from (3) and (4) that E(δ^R2)+E(δ^S2)=δR2+δS2.Table 1
Performance of estimators ofδ^R2,δ^S2,δ^T2 for several values of ga and na. The true values are δR2=.022,δS2=.012,δT2=0.022, and negative estimates of δ^S2 are allowed. The RMSEs for g =3, n =16 agree with that predicted by (6) for merging successive groups, and the RMSEs for g =12, n = 4 agree with that predicted by (7) for splitting each group into 2 groups.
Percent relative bias inδ^R2,δ^S2,δ^T2 Percent relative RMSE inδ^R2,δ^S2,δ^T2 g a =6,na =8 (correct) 0, 0, 0 22, 96, 24  g a =2, na =24 8, -45, 0 32,99,25  g a =24, na =2 0, -7, -1 29,114,23  g a =3,na =16 7, -32, 0 29,92,25  g a =12,na =4 0,-5,-1 24,95,23  MAD,FAP=0.1 0,5,1 25,103,25  MAD, FAP=0.05 3,-6,1 26,101,26  MAD, FAP=0.01 1, 0, 1 25,102,25As another extreme case, assume that each of the 6 true groups in Figure3 is split into 3 groups, creating 18 groups of size 2. It is easily shown for this case that E(δ^R2)=δR2 and E(δ^S2)=(15/17)δS2. The factor 15/17 arises from having 3 repeated values of S1,S2,…, and S6 in the expression(5)En∑j=1gd-j∙-d-∙∙2g-1=E2∑j=118d-j∙-d-∙∙217=217E∑j=118d-j∙2-E∑j≠j′d-j∙d-j′∙18×17=2δS2+δR22-6×618×17δS2=δR2+2×1517δS2.More generally,(6)Ena∑i=1gd-i∙-d-∙∙2ga-1=naδS2+δR2na-gtntnt-1gaga-1δS2and so(7)Eδ^S2=1-gtnt/nant/na-1gaga-1δS2.The extreme case of merging distinct groups 1 and 2, 3 and 4, and 5 and 6 or of splitting each of the 6 true groups into 3 groups will not occur in this simulated data or in other similar data if any reasonably effective change detection method were used, as illustrated in Section 4.


## 4. Change Detection to Detect Changes in the Value of S =SI-SO

There are many change detection options for sequences of (O-I) data [9, 10]. Assuming that not too many changes occur in S=SO+SI during the entire analysis period, a robust estimate of δR, where R=RO+RI, is available, using the median absolute deviation (MAD) of the lag-one differences, adjusted to provide an unbiased estimate of the true variance [8]. The following function madwin implements the MAD-based estimate of δR for a moving window change detector in the R statistical programming language [8]. The moving window compares (d-j-d-j+1) to kδ^R, where d-j is the average value of d=(O-I)/O in window j (which by the central limit theorem can be assumed to have approximately a normal distribution) and k is a threshold chosen such that the false alarm probability over the entire analysis period is small, such as 0.05.madwin = function(x=testdata1-testdata2,nwin1=5,nwin2=7,t.percentile=.99,peak.threshold=.5)# nwin1 is the width of the moving window;#nwin2 is the width of a second window that is used to eliminate duplicate inferred change points #arising from the same mean shift (shift inS value).​​n = length(x); temp.check = numeric(n); indices.change = 0tempdiff =c(0,diff(x))rstd = mad(diff(x)); rvar =rstd∧2​​# the mad function provides a robust used of the standard deviation# that is adjusted to have zero bias for Gaussian data.df1.use =mean(c(n-1,nwin1));df2.use = mean(c(n-1,nwin2)) # approximate degrees of freedomfor(i in (nwin1+1):(n-nwin1+1))# the first candidate change point is at index nwin1+1indicesl = (i-nwin1):(i-1);indicesh = i:(i+nwin1-1)tempdiff = mean(x[indicesh]) - mean(x[indicesl]) # difference between left and right windowstemp.check[i] = tempdiff/(rstd/nwin1∧.5)if(abs(temp.check[i]) > qt(t.percentile,df=df1.use)) indices.change = c(indices.change,i)indices.change = indices.change[-1]indices.change1 = 0​​temp = abs(temp.check)for(i in (nwin2+1):(n-nwin2-1))if(temp[i] == max(temp[(i-nwin2):(i+nwin2)]) && abs(temp.check[i]) > qt(t.percentile,df=df2.use)) indices.change1 = c(indices.change1,i)indices.change1[-1] # function returns indices.change1, discarding the initial 0.Figure4 illustrates a moving window option and a cumulative sum option [3, 10] (monitoring separately for positive shifts and for negative shifts) that both use the MAD-based estimate of δR. The data is simulated with g = 4, n = 100, δR=δRO2+δRI2=0.0022, and δS=δSO2+δSI2=0.011. The estimated number of groups g^ is a random quantity that varies across hypothetical realizations of the data for the same fixed unknown true grouping. In simulation studies, one can know the true grouping and enforce all assumptions to hold, so it is possible to compare the performance of δ^R2 and δ^S2 for candidate grouping options.Figure 4
A plot of results of two change detection option s (moving average difference = “2” and Page’s cumulative sum = “3” for positive shift detection and “4” for negative shift detection).In Figure4, there are three change points, at 101, 201, and 301, so there are four true groups. The black 1s are the d values. Notice the positive mean shift from group 1 (1:100) to group 2 (101:201). The red 2s are the moving average difference, scaled by using the MAD-based robust estimate of δR. The blue and green curves are Page’s cumulative sum, defined as Pi=max⁡(0,Pi-1+d~i), where d~=d/σ^robust,d,random [3, 10] for positive and for negative change detection. The red 2s (size 3 moving window) easily detect all three changes, at 101, 201, and 301. The cusums are more difficult to assess visually, but correctly have edges near the change points. The two horizontal lines are approximate 0.95 false alarm probability limits for the parametric size 3 moving window using the MAD approach, so the moving window option has a few false alarms where the red 2s exceed the horizontal lines away from the three change points, meaning it detects more than four groups in this realization. The moving window option computes the difference between successive moving averages of, for example, 3, (user specified) successive nonoverlapping d~ values.Figure5 is a simulated example to illustrate that even if δS/δR is large, so that most mean shifts due to S=SO+SI shifting will be easily observed, some shifts will be small and difficult to detect. Similarly, in Figure 5, the mean shift from group 1 to group 2 might not be detected, particularly if δR increased.Figure 5
Even withδR=0.01,δS=0.3 so δS/δR=30, some group changes will not necessarily be detected (such as the change from group 1 to group 2).Other change detection options can be considered; for example, a nonparametric option has been implemented that works with the ranks ofd values in a moving window system similar to the parametric window system used here. Nonparametric options can outperform parametric options if the distributional assumptions made by the parametric options are badly violated in the data.Several simulations have been performed in R [8] to assess the impacts of having to perform change detection to choose the groups. As shown above, incorrect group assignments can lead to negative bias in δ^S2 and to positive bias in δ^R2. The IAEA’s calculations of the sample size needed to attain a specified loss detection probability require separate estimation of δ^S and δ^R [5].For the results in Tables1 and 2, the simulations used a total of 48 observations, divided into cases defined as having 2, 3, 4, 6, 8, 12, 16, or 24 equal-sized groups of 24, 16, 12, 8, 6, 4, 3, or 2, observations, respectfully. Each case was analysed using 105 simulations, and for brevity here, only some of the results are reported in Tables 1 and 2. The Table 1 results (repeatable to ±1 across sets of 105 simulations) are the relative bias (in percent) and the relative root mean-squared error (in percent). Table 2 is the same, but negative values of δ^S2 are set to 0, and the Table 2 entries are for estimation of δR,δS,δT, and δS rather than for estimation of δR2,δS2,δT2. The RMSE is defined in the usual manner, for example, for δ^R2 in Table 1 as RMSE=∑i=1nsim(δ^Ri2-δR2)2/nsim, and the relative RMSE is defined as RMSE/δR2. The results reported in Tables 1 and 2 use the correct grouping (g = 6, n=8) and several wrong groupings as indicated. Tables 1 and 2 also use the MAD-based windowing change detection method (with a false alarm probability, FAP, of 0.1, 0.05, or 0.01 as listed) and reduce the bias in the estimates δ^S2 or δ^S. The RMSE in δ^S2 or δ^S is only slightly larger than in the case where the true groups are known.Table 2
Performance of estimators ofδ^R,δ^S,δ^T for several values of ga and na. The true values are δR=.02,δS=.01,δT=0.022, and negative estimates of δ^S2 were set to 0.
Percent relative bias inδ^R,δ^S,δ^T Percent relative RMSE inδ^R,δ^S,δ^T g a =6,na =8 (correct) 0, 0, 0 11, 52, 12  g a =2,na =24 8, -40, 0 15, 78, 12  g a =24,na =2 0, 0, 0 14, 60, 12  g a =3,na =16 6, -28, 0 13,68,12  g a =12,na =4 0,-2,0 12,54,12  MAD, FAP=0.1 13, 6,1 12,53,12  MAD, FAP=0.05 3,-4,1 12, 58, 12  MAD, FAP=0.01 1, 2, 1 12,55,12Figure6(a) plots an example realization of the data with g = 6, n = 8 and with δR=.02,δS=.01. Figure 6(b) plots the results of the MAD-based windowing change detection method with a false alarm probability (deciding that a potential change point is a true change point when in fact it is not) of 0.01. Figure 7 is the same as Figure 6, but for with δR=.02,δS=.06. Based on the 105 simulations, the distribution of the number of groups found using the false alarm rate of 0.05, 0.17, 0.33, 0.30, 0.13, 0.02, 0.002, and 0.00006, for ga = 4,5,6,7,8,9,10, or 11, respectively; the average number of groups inferred is 5.4; the standard deviation in the estimated number of groups is 1.1.Figure 6
Example of change detection results using a sliding window and the MAD-based estimate ofδR. The simulated data had g = 6 groups, each with n = 8 observations, with δR=.02,δS=.01.
(a)
True groups (b)
Inferred groupsFigure 7
Example of change detection results using a sliding window and the MAD-based estimate ofδR. The simulated data had g = 6 groups, each with n = 8 observations, with δR=.02,δS=.06.
(a)
True groups (b)
Inferred groupsFigure8 plots real (O-I)/O with the groups being the inspection periods in Figure 8(a) and the groups being inferred using windowing in Figure 8(b). In Figure 8(a), the estimates are δ^R=.050,δ^S=0,δ^T=0.051. In Figure 8(b), the estimates are δ^R=.050,δ^S=0.01,δ^T=0.051. Figure 9 plots a second example of real (O-I)/O with the groups being the inspection years in Figure 9(a), inspection periods (multiple inspection periods year year) in Figure 9(b), and the groups being inferred using windowing in Figure 8(c). In Figure 9(a), the estimates are δ^R=.050,δ^S=0.010,δ^T=0.051. In Figure 9(b), the estimates are δ^R=.051,δ^S=0.017,δ^T=0.053. In Figure 9(c), the estimates are δ^R=0.47,δ^S=0.028,δ^T=0.054.Figure 8
Real (O-I)/O data with the groups being the inspection periods in (a) and the groups being inferred using windowing in (b).
(a)
Groups are inspection periods (b)
Groups are inferred using windowingFigure 9
Real (O-I)/O data with the groups being the inspection years in (a), inspection periods in (b) (with multiple inspection periods per year), and the groups being inferred using windowing in (c).
(a)
Groups are inspection years (b)
Groups are inspection periods (c)
Groups are inferred using windowing


## 5. Discussion and Summary

This paper introduced a lag-one differencing method to estimate the random error standard deviationδR and then used the estimate δ^R to calculate a change detection threshold in a moving window method to detect shifts in the short-term systematic error. Performance results on simulated and real data were presented. The impact of having to perform change detection on the estimated short-term systematic and random error variances appears to be modest or small. In the motivating example, typically, it is assumed that short-term systematic errors change across inspection periods, so inspection periods form the groups used in the ANOVA. In some data sets, it appears that the short-term errors have changed at other times, so change detection methods could be used to detect the actual change times.The default assumption that is often adequate is thatS changes across inspection periods, so the groups are defined as inspection periods. However, if change detection contradicts the default assumption that S changes across inspection periods, then change detection as presented in Section 4 is a reasonable option. Recall from Section 3 that E∑j=1g∑k=1n(djk-d-∙∙)2/(ng-1)=δR2+δS2(ng-n)/(ng-1), so if the grouping structure were ignored totally (implicitly assuming g = 1), then the sample variance ∑j=1g∑k=1n(djk-d-∙∙)2/(ng-1) has expected value δR2+δS2(ng-n)/(ng-1). For example, with ga = 1, na = 48, gt = 6, nt = 8 (Section 4 example), then the expected value of ∑j=1g∑k=1n(djk-d-∙∙)2/(ng-1) is δR2+40δS2/47, which is not dramatically different from the desired valueδR2+δS2. Fortunately (see Tables 1 and 2), the bias in δ^S2 is not that much larger when the groups are chosen by change detection than in the known-groups case. Interestingly, due to a negative bias in δ^S2, the RMSE in δ^S2 can be slightly smaller for some wrong-groups choices than for the known-groups case. However, negative bias in δ^S2 could lead to more false alarms in nuclear safeguards applications, such as the application illustrated in Figure 1 [5], so it is preferable to use change detection to avoid large negative biases. Any reasonably effective change detection method [9, 10] should be adequate in this context; the windowing scheme based on a robust MAD-based estimate of δ^R is among the simplest reasonable options. References [2–5] indicate how the error variances δR2 and δS2 propagate in several examples; generally, δS2 tends to be smaller than δR2 but often has larger impact on the variance of the difference statistic D=(N/n)∑j=1n(Oj-Ij) or of the material balance MB=Transferin+InvB-InvE-Transferout [2–5]. Therefore, grouping choices that lead to negative bias in δ^S2 will tend to underestimate variance (D) and variance(MB).Data such as that in Figure1(a) are collected in statistical quality control applications, where in phase I control limits are estimated, and in phase II the process is monitored using the estimated limits from phase I [11]. A key choice in quality control is the choice of rational subgroups [11]; a good rational subgrouping minimizes variation within groups and maximizes variation between groups. Ideally, only random errors are present within groups, while other effects might be present between groups. Analysis of Phase I data is retrospective, such as presented here for paired (O,I) data, where the windowing scheme based on a robust MAD-based estimate of δ^R was introduced to find good subgroups such as those found in Figure 7 (the inferred groups closely approximated the true groups).One open question is to estimate the RMSE inδ^S for any particular data set for which the true values of δR and δS are unknown. One option is to use the same type of simulations that were used to estimate the bias and RMSE in δ^S (Tables 1 and 2), using δ^R and δ^S as the true values of δR and δS. However, this approach is likely to be quite noisy unless the number of groups g is large enough that the RMSE in δ^S is small. The relative RMSE of 52% in δ^S is too large (Table 2) for g = 6, n = 8, so more than 6 groups would probably be needed for reliable estimation of the RMSE in δ^S. A second open question is to estimate the actual false alarm probability using the estimates δ^S and δ^R in data such as that in Figure 1, when the nominal false alarm probability is specified, such as 0.05. Reference [12] addressed this question using tolerance intervals assuming that the group memberships are known. A third open question is to evaluate the impact of having to infer group memberships on δ^RO,δ^SO,δ^RI,δ^SI.



---

*Source: 1769149-2019-01-01.xml*