Generalized Degrees of Freedom (GDF)

Two popular regression methods for the multiplicative-error model are the MinimumUnbiased-Percent Error (MUPE) and Minimum-Percentage Error under the Zero-Percentage Bias (ZMPE) methods. The MUPE method, an Iteratively Reweighted Least Squares (IRLS) regression, does not use any constraints, while the ZMPE method requires a constraint as part of the curve-fitting process. However, ZMPE users do not adjust the degrees of freedom (DF) to account for constraints included in the regression process. As a result, fit statistics for the ZMPE equations, e.g., the standard percent error (SPE) and generalized R (GRSQ), can be incorrect and misleading. This results in incomparable fit statistics between ZMPE and MUPE. This paper details why DF should be adjusted and recommends a Generalized Degrees of Freedom (GDF) measure to calculate fit statistics for constraint-driven cost estimating relationships (CER). It also explains why ZMPE’s standard error underestimates the spread of the CER error distribution. Illustrative examples are provided. OUTLINE This paper introduces a new concept to the cost community. We explain in detail why degrees of freedom (DF) should be adjusted for any constraint-driven regression equations and we use ZMPE as an example. The following topics will be discussed: • Purpose • Using constraints in CER development • Additive and Multiplicative-Error Models • ZMPE vs. MUPE:  Is ZMPE CER Unbiased?  Is ZMPE’s Standard Percent Error (SPE) Unbiased? • Definition of Generalized Degrees of Freedom (GDF) • Calculate Fit Statistics Using GDF • Example Section • Conclusions • Future Study Items for Constraint Driven Regression PURPOSE The main objectives of this paper are threefold. First, we address a potential shortcoming when deriving constraint-driven equations, e.g., ZMPE CERs. Second, we suggest an appropriate adjustment (GDF measure) to calculate the CER fit statistics when constraints are included in the Tecolote Research, Inc. PRT-191, Generalized Degrees of Freedom (GDF) Page 2 regression process. Without a proper adjustment for DF, the fit measures for the ZMPE equations, e.g., the standard percent error (SPE) and generalized R (GRSQ), can be incorrect and misleading. This results in an unfair comparison with the statistical measures derived by the MUPE method. Lastly, we explain why using ZPME CERs without adjustment in cost uncertainty analysis may unduly tighten the S-curve. The unadjusted SPE for the ZMPE equation is smaller than the SPE for the MUPE equation. Since the SPE measures are used as the basis for cost uncertainty analysis, using unadjusted SPE measures for ZMPE equations may have a direct impact on the cost uncertainty results. A math proof is provided to support this conclusion. USING CONSTRAINTS IN CER DEVELOPMENT Solver (an Excel add-in program) is a popular tool used to generate nonlinear CERs, especially when constraints are specified. Many analysts have been using the following methodologies in Solver to develop CERs: • Minimizing the sum of squared residuals under the Zero-Percentage Bias constraint (i.e., the mean of the percentage errors is zero) • Minimizing the sum of squared percentage errors under the Zero-Percentage Bias constraint (i.e., the ZMPE CER) • Minimizing the sum of squared percentage errors or residuals in log space under the Zero-Bias constraint (i.e., the mean of the residuals is zero) using the BalanceAdjustment Factor (BAF) (see Book, 2006) Considerations. These methodologies are very different from traditional methods, such as ordinary least squares (OLS) or log-linear models (OLS performed in log space). When various constraints are specified, we may not have the DF as given by the traditional definition when constraints are not part of the curve fitting process. In fact, the DF should be reduced if constraints are specified when deriving CERs. (See the explanation in the definition section.) In addition, constraints should be made consistent with the underlying hypothesis. Mixing a multiplicative constraint with an additive error model or vice versa is neither logical nor necessary. Suggestions. Here are a few suggestions when using constraints in Solver: • Do not abuse Solver. Although Solver can easily handle constraints in the minimization process, we should not specify excessive constraints. • Explore different starting points to see if the solution stabilizes when using Solver. As we know, Solver is sensitive to the starting points and can be easily trapped in local minima, especially when fitting complicated equations or the sample size is small. We also know that the ZMPE CERs seem to be less stable than the MUPE ones (see Hu and Smith, 2007). Be cautious when fitting ZMPE equations to small samples. As a precaution, explore different starting points to see if they lead to the same solution to ensure the resultant solution is stable. • Specify “meaningful” constraints. Make sure constraints included in the process are necessary, logical, and statistically sound as DF can be reduced by additional constraints. Tecolote Research, Inc. PRT-191, Generalized Degrees of Freedom (GDF) Page 3 If we lose one DF for each constraint, we want to ensure it is worthwhile. Since we generally deal with small samples, we should preserve as many DF as we can. • Calculate the DF and fit statistics properly if constraints are specified when deriving CERs, as well as any estimating relationships. ADDITIVE AND MULTIPLICATIVE ERROR MODELS We first introduce additive and multiplicative error models and then explain three different methodologies for fitting multiplicative error models. Additive Error Model. An additive error model is generally stated as follows: i i i f Y ε + = ) , ( β x = fi + εi for i = 1, ., n (1) where: Yi = observed cost of the i data point, i = 1 to n f (xi,β) = fi = the value of the hypothesized equation at the i data point xi = vector of the cost driver variables at the i data point β = vector of coefficients (unknown parameters) to be estimated by the regression equation εi = error term with a mean of 0 and variance σ (assumed to be independent of the cost drivers) n = sample size The corresponding standard error of estimate for an additive CER is commonly termed the standard error of estimate (SEE) or CER’s standard error: