Additional Analyses of the Monte Carlo 

Model Developed for the Determination 

of PEMS Measurement Allowances for 

Gaseous Emissions Regulated Under 

the Heavy-Duty Diesel Engine In-Use 

Testing Program 
Additional Analyses of the Monte Carlo 


Model Developed for the Determination 


of PEMS Measurement Allowances for 


Gaseous Emissions Regulated Under 


the Heavy-Duty Diesel Engine In-Use 


Testing Program


Assessment and Standards Division 

Office of Transportation and Air Quality 

U.S. Environmental Protection Agency 

Prepared for EPA by


Southwest Research Institute


EPA Contract No. EP-C-05-018


Work Assignment No. 2-6


NOTICE 

This technical report does not necessarily represent final EPA decisions or 

positions. It is intended to present technical analysis of issues using data 

generated in the associated test program. The purpose in the release of such 

reports is to facilitate the exchange of technical information and to inform the 

public of technical developments which may form the basis for a final EPA 

decision, position, or regulatory action. 

EPA420-R-07-010 

August 2007 
EXECUTIVE SUMMARY 
 

This report documents the program conducted by Southwest Research Institute® (SwRI), 

on behalf of the U.S. Environmental Protection Agency (EPA), the objective of which was to 

perform additional analyses on the Portable Emissions Measurement Systems (PEMS) Monte 

Carlo simulation models in order to determine validation of the three emissions (BSNOX, 

BSNMHC and BSCO) using three calculation methods. 

Several steady-state error surfaces were modified based on recommendations from the 

Heavy-Duty In-Use Testing (HDIUT) Steering Committee. These included modifications to the 

steady-state NOX, exhaust flow rate, CO and CO2 error surfaces. Several reference NTE events 

were run which produced validation results from all three emissions and all three calculation 

methods. 

Measurement allowances were computed for two of the simulation strategies based on 

using 50 reference NTE events. The Mod 1 strategy involved changing the steady-state CO, CO2 

and exhaust flow rate error surfaces to eliminate bias while changing the steady-state NOX to a 

level independent error surface including all the test data. The Mod 2 strategy was the same as 

Mod 1 except the steady-state NOX error surface was also changed to a level independent error 

surface but excluded several questionable low NOX values from one of the test engines. The 

measurement allowance values by calculation method determined at the conclusion of these 

analyses are summarized in Table 1. This report details the process used to determine the 

measurement values reported in Table 1. 

TABLE 1. MEASUREMENT ALLOWANCES FOR MOD 1 AND MOD 2 

Pollutant Calculation 

Method 

Mod 1 Measurement 

Allowance, g/hp-hr 

Mod 2 Measurement 

Allowance, g/hp-hr 

NOX 1 0.232 0.211 

2 0.178 0.151 

3 0.192 0.171 

NMHC 1 0.015 0.014 

2 0.014 0.014 

3 0.014 0.014 

CO 1 0.268 0.266 

2 0.258 0.250 

3 0.270 0.262 

SwRI Report 03.12859.06 i 
LIST OF ACRONYMS 

Brake-Specific BS 

California Air Resources Board CARB 

Center for Environmental Research 

& Technology CE-CERT 

Code of Federal Regulations CFR 

Electronic Flow Meter EFM 

Empirical Distribution Function EDF 

Engine Control Module ECM 

Engine Manufacturer’s Association EMA 

Environmental Protection Agency EPA 

Heavy Duty In-Use Testing HDIUT 

Heavy Heavy Duty HHD 

Light Heavy Duty LHD 

Median Absolute Deviation MAD 

Medium Heavy Duty MHD 

Memorandum of Agreement MOA 

Mobile Emissions Laboratory MEL 

Not To Exceed NTE 

Portable Emissions Measurement System PEMS 

Root Mean Square RMS 

SEMTECH-DS SN G05-SDS04 PEMS 1 

SEMTECH-DS SN G05-SDS02 PEMS 2 

SEMTECH-DS SN G05-SDS03 PEMS 3 

SEMTECH-DS SN G05-SDS01 PEMS 4 

SEMTECH-DS SN D06-SDS01 PEMS 5 

SEMTECH-DS SN D06-SDS06 PEMS 6 

SEMTECH-DS SN F06-SDS02 PEMS 7 

Southwest Research Institute SwRI 

Standard Deviation SD 

SwRI Report 03.12859.06 ii 
TABLE OF CONTENTS 

Page

EXECUTIVE SUMMARY ............................................................................................................. i

LIST OF ACRONYMS .................................................................................................................. ii 

LIST OF FIGURES ....................................................................................................................... iv 

LIST OF TABLES......................................................................................................................... vi 

1.0 INTRODUCTION .............................................................................................................. 1

2.0 ERROR SURFACE MODIFICATIONS............................................................................ 2 

2.1 Steady-State Exhaust Flow Rate Error Surface .................................................................. 2 

2.2 Steady-State CO2 Error Surface.......................................................................................... 3 

2.3 Steady-State CO Error Surface ........................................................................................... 4 

2.4 Steady-State NOx Error Surface.......................................................................................... 5 

3.0 REFERENCE NTE EVENTS............................................................................................. 9 

4.0 CE-CERT MEASUREMENTS ........................................................................................ 13 

5.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS 

USING MOD 1, MOD 2 AND MOD 3............................................................................ 14 

6.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS 

USING MOD B ................................................................................................................ 25 

7.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS 

USING MOD 1................................................................................................................. 35 

8.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS 

USING MOD 2................................................................................................................. 45 

9.0 MEASUREMENT ALLOWANCE CALCULATIONS .................................................. 56 

10.0 VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS.... 77 

11.0 FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS ... 81 

12.0 REFERENCES ................................................................................................................. 83 

0

SwRI Report 03.12859.06 iii 
LIST OF FIGURES 

Figure Page

1 Revised Error Surface for Steady-State Exhaust Flow Rate....................................3 

2 Revised Error Surface for Steady-State CO2 ...........................................................4 

3 Revised Error Surface for Steady-State CO.............................................................5 

4 Revised Error Surface Mod 1 for Steady-State NOx ...............................................6 

5 Revised Error Surface Mod 2 for Steady-State NOx ...............................................7 

6 Revised Error Surface Mod 3 for Steady-State NOx ...............................................8 

7 Distribution of Ideal NOx (g/kW-hr) Method 1 for 23 Selected Ref NTE 

Events and 195 Ref NTE Events ...........................................................................11 

8 Distribution of Ideal NOx (g/kW-hr) Method 1 for 50 Selected Ref NTE 

Events and 195 Ref NTE Events ...........................................................................12 

9 Validation EDF Plots Using 23 Reference NTE Events for NOx Method 1 

Mods 1, 2 and 3......................................................................................................16 

10 Validation EDF Plots Using 23 Reference NTE Events for NOx Method 2 for 

Mods 1, 2 and 3......................................................................................................17 

11 Validation EDF Plots Using 23 Reference NTE Events for NOx Method 3 for 

Mods 1, 2 and 3......................................................................................................18 

12 Validation EDF Plots Using 23 Reference NTE Events for NMHC Method 1 

for Mods 1, 2 and 3................................................................................................19 

13 Validation EDF Plots Using 23 Reference NTE Events for NMHC Method 2 

for Mods 1, 2 and 3................................................................................................20 

14 Validation EDF Plots Using 23 Reference NTE Events for NMHC Method 3 

for Mods 1, 2 and 3................................................................................................21 

15 Validation EDF Plots Using 23 Reference NTE Events for CO Method 1 for 

Mods 1, 2 and 3......................................................................................................22 

16 Validation EDF Plots Using 23 Reference NTE Events for CO Method 2 Mods 

1, 2 and 3................................................................................................................23 

17 Validation EDF Plots Using 23 Reference NTE Events for CO Method 3 for 

Mods 1, 2 and 3......................................................................................................24 

18 Validation EDF Plots Using 23 Reference NTE Events for NOx Method 1 Mod B.....26 

19 Validation NTE Events Using 23 Reference NTE Events for NOx Method 2 Mod B..27 

20 Validation EDF Plots Using 23 Reference NTE Events for NOx Method 3 Mod B.....28 

21 Validation EDF Plots Using 23 Reference NTE Events for NMHC Method 1 Mod B..29 

22 Validation EDF Plots Using 23 Reference NTE Events for NMHC Method 2 Mod B..30 

23 Validation EDF Plots Using 23 Reference NTE Events for NMHC Method 3 Mod B..31 

24 Validation EDF Plots Using 23 Reference NTE Events for CO Method 1 Mod B......32 

25 Validation EDF Plots Using 23 Reference NTE Events for CO Method 2 Mod B......33 

26 Validation EDF Plots Using 23 Reference NTE Events for CO Method 3 Mod B......34 

27 Validation EDF Plots Using 50 Reference NTE Events for NOx Method 1 Mod 1 .....36 

28 Validation EDF Plots Using 50 Reference NTE Events for NOx Method 2 Mod 1 .....37 

29 Validation EDF Plots Using 50 Reference NTE Events for NOx Method 3 Mod 1 .....38 

30 Validation EDF Plots Using 50 Reference NTE Events for NMHC Method 1 Mod 1...39 

31 Validation EDF Plots Using 50 Reference NTE Events for NMHC Method 2 Mod 1...40 

32 Validation EDF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 1...41 

33 Validation EDF Plots Using 50 Reference NTE Events for CO Method 1 Mod 1.......42 

34 Validation EDF Plots Using 50 Reference NTE Events for CO Method 2 Mod 1.......43 

SwRI Report 03.12859.06 iv 
35 Validation EDF Plots Using 50 Reference NTE Events for CO Method 3 Mod 1.......44 

36 Validation EDF Plots Using 50 Reference NTE Events for NOx Method 1 Mod 2 .....46 

37 Validation EDF Plots Using 50 Reference NTE Events for NOx Method 2 Mod 2 .....47 

38 Validation EDF Plots for 50 Reference NTE Events for NOx Method 3 Mod 2...48 

39 Validation EDF Plots for 50 Reference NTE Events for NMHC Method 1 Mod 2.....49 

40 Validation EDF Plots for 50 Reference NTE Events for NMHC Method 2 Mod 2.....50 

41 Validation EDF Plots Using 50 Reference NTE Events for NMHC Method 3 Mod 2...51 

42 Validation EDF Plots Using 50 Reference NTE Events for CO Method 1 Mod 2.......52 

43 Validation EDF Plots Using 50 reference NTE Events for CO Method 2 Mod 2.53 

44 Validation EDF Plots Using 50 Reference NTE Events for CO Method 3 Mod 2.......54 

45 Regression Plot of 95th Percentile Delta BSNOx Versus Ideal BSNOx for Method 1 

Mod 1......................................................................................................................57 

46 Regression Plot of 95th Percentile Delta BSNOx Versus Ideal BSNOx for Method 2 

Mod 1......................................................................................................................58 

47 Regression Plot of 95th percentile Delta BSNOx Versus Ideal BSNOx for Method 3 

Mod 1......................................................................................................................59 

48 Regression Plot of 95th percentile Delta BSNMHC Versus Ideal BSNMHC for 

Method 1 Mod 1 ......................................................................................................60 

49 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for 

Method 2 Mod 1 ......................................................................................................61 

50 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for 

Method 3 Mod 1 ......................................................................................................62 

51 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 1 

Mod 1 .....................................................................................................................63 

52 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 2 

Mod 1 .....................................................................................................................64 

53 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 3 

Mod 1 .....................................................................................................................65 

54 Regression Plot of 95th Percentile Delta BSNOX Versus Ideal BSNOX for Method 1 

Mod 2......................................................................................................................66 

55 Regression Plot of 95th Percentile Delta BSNOx Versus Ideal BSNOx for Method 2 

Mod 2......................................................................................................................67 

56 Regression Plot of 95th Percentile Delta BSNOx Versus Ideal BSNOx for Method 3 

Mod 2......................................................................................................................68 

57 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for 

Method 1 Mod 2 ......................................................................................................69 

58 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for 

Method 2 Mod 2 ......................................................................................................70 

59 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for 

Method 3 Mod 2 ......................................................................................................71 

60 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 1 

Mod 2 .....................................................................................................................72 

61 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 2 

Mod 2 .....................................................................................................................73 

62 Regression Plot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 3 

Mod 2 .....................................................................................................................74 

0

SwRI Report 03.12859.06 v 
LIST OF TABLES 
 

Table Page 

1 Measurement Allowances For MOD 1 and Mod 2.................................................. i 
 

2 Revised Error Surfaces Used in Monte Carlo Simulations......................................8 
 

3 Reference NTE Events Used in Monte Carlo Simulations....................................10 
 

4 Descriptive Statistics for Ideal BSNOx (g/kW-hr) for Various NTE Subsets .......12 
 

5 Convergence criteria by Emission .........................................................................14 
 

6 BSNOx Validation Results Based on 23 Reference NTE Events ..........................15 
 

7 Summary of Validation Results.............................................................................55 
 

8 Measurement Error at Threshold for BSNOx Using Regression and Median 
 

Methods for Method 1 Mod 1................................................................................57 
 

9 Measurement Error at Threshold for BSNOx Using Regression and Median 
 

Methods for Method 2 Mod 1................................................................................58 
 

10 Measurement Error at Threshold for BSNOx Using Regression and Median 
 

Methods for Method 3 Mod 1................................................................................59 
 

11 Measurement Error at Threshold for BSNMHC Using Regression and Median 
 

Methods for Method 1 Mod 1................................................................................60 
 

12 Measurement Error at Threshold for BSNMHC Using Regression and Median 
 

Methods for Method 2 Mod 1................................................................................61 
 

13 Measurement Error at Threshold for BSNMHC Using Regression and Median 
 

Methods for Method 3 Mod 1................................................................................62 
 

14 Measurement Error at Threshold for BSCO Using Regression and Median 
 

Methods for Method 1 Mod 1................................................................................63 
 

15 Measurement Error at Threshold for BSCO Using Regression and Median 
 

Methods for Method 2 Mod 1................................................................................64 
 

16 Measurement Error at Threshold for BSCO Using Regression and Median 
 

Methods for Method 3 Mod 1................................................................................65 
 

17 Measurement Error at Threshold for BSNOx Using regression and Median 
 

methods for Method 1 Mod 2 ................................................................................66 
 

18 Measurement Error at Threshold for BSNOx Using Regression and Median 
 

methods for Method 2 Mod 2 ................................................................................67 
 

19 Measurement Error at Threshold for BSNOx Using Regression and Median 
 

Methods for Method 3 Mod 2................................................................................68 
 

20 Measurement Error at Threshold for BSNMHC Using Regression and Median 
 

Methods for Method 1 Mod 2................................................................................69 
 

21 Measurement Error at Threshold for BSNMHC Using Regression and Median 
 

Methods for Method 2 Mod 2................................................................................70 
 

22 Measurement Error at Threshold for BSNMHC Using Regression and Median 
 

Methods for Method 3 Mod 2................................................................................71 
 

23 Measurement Error at Threshold for BSCO Using Regression and Median 
 

Methods for Method 1 Mod 2................................................................................72 
 

24 Measurement Error at Threshold for BSCO Using Regression and Median 
 

Methods for Method 2 Mod 2................................................................................73 
 

SwRI Report 03.12859.06 vi 
25 Measurement Error at Threshold for BSCO Using Regression and Median 

Methods for Method 3 Mod 2................................................................................74 

26 Summary of Measurement Errors at Respective Threshold (%) ...........................75 

27 Summary of Measurement Allowance, g/hp-hr.....................................................76 

28 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte 

Carlo validation Simulations for BSNOx ...............................................................78 

29 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte 

Carlo validation Simulations for BSNMHC..........................................................79 

30 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte 

Carlo validation Simulations for BSCO ................................................................80 

31 Sensitivity Results Comparing 13 Reference NTE Events Across 5 Monte 

Carlo full model Simulations for BSNOx ..............................................................82 

SwRI Report 03.12859.06 vii 
1.0 INTRODUCTION 

Southwest Research Institute completed a Portable Emissions Measurement System 

(PEMS) program on behalf of the U.S. Environmental Protection Agency (EPA) in early April 

2007. The purpose of this project was to determine the brake-specific (BS) measurement 

allowances for the gaseous pollutants regulated under the Heavy-Duty In-Use Testing (HDIUT) 

program [Ref 1]. The study was performed under cooperation between the EPA, the California 

Air Resources Board (CARB), and the Engine Manufacturer’s Association (EMA). All efforts 

during this program were conducted under the direction of a joint body, the HDIUT 

Measurement Allowance Steering Committee, referred to in this report simply as the Steering 

Committee. 

The program consisted of modeling various PEMS measurement errors using a statistical 

Monte Carlo modeling simulation. The simulation results were used to generate the brakespecific measurement allowances based on three calculation methods for BS emissions of NOx, 

NMHC and CO. To confirm the results of the simulation, a SEMTECH-DS PEMS was operated 

in-use with the CE-CERT Mobile Emission Laboratory (MEL) over several routes in California. 

The differences between the PEMS and MEL gaseous emission measurements were used to 

validate the SwRI Monte Carlo modeling simulation results. 

Calculation Methods 2 (BSFC based) and 3 (ECM Fuel Specific) did not validate for 

BSNOx based on the CE-CERT MEL emissions and the Monte Carlo PEMS simulations. Given 

this result, EPA contracted with SwRI in April 2007 to conduct additional analyses to identify 

possible causes as to why the BSNOx did not validate. The results of these analyses are included 

in this report. 

SwRI Report 03.12859.06 1 of 83 
2.0 ERROR SURFACE MODIFICATIONS 

Since calculation Methods 2 and 3 did not validate for BSNOx after the Monte Carlo 

model simulation runs during the original PEMS study, the associated Steering Committee chose 

to modify key error surfaces in order to determine if such alterations would effect the validation

of Methods 2 and 3. Over the course of several meetings, the Steering Committee brainstormed 

to generate alternate error surface processing methods. The Committee decided to modify 

Steady-State CO, CO2, Exhaust Flow Rate, and NOx, as these surfaces had significant influence 

on the Model results based on the sensitivity analyses. 

In reprocessing the error surfaces, the Steering Committee agreed that it would be 

necessary to remove any biases that were recorded in the SwRI laboratory. It was assumed that 

these bias errors were due to the limited number of PEMS units and comparative observations, 

and that if more PEMS and Sensors Inc. exhaust flow meters were tested the biases may have 

been eliminated. Therefore, to remove the biases the original steady-state error surfaces were 

transformed to be a symmetric error surface with the 50th percentile errors set to zero and the 5th

and 95th percentile errors modified to be a mirror image of one another. Two analysis methods

were used to generate the revised error surfaces. If the recorded error data showed an emissions

level dependency, an envelope was generated to encompass the extreme 5th or 95th percentile

error data. The envelope segments with the largest absolute error were then mirrored to generate 

a symmetric error surface. A more rigorous analysis was used to reprocess level independent

error surfaces. 

2.1 Steady-State Exhaust Flow Rate Error Surface 

Figure 1 shows the reprocessed, level-dependent steady-state exhaust flow rate error 

surface as well as the exhaust flow error surface data that was used in the original MC model 

runs. The largest errors recorded during laboratory testing were positive deltas measured during 

Engine 3 testing with the 3-inch EFMs. The 3-inch EFM errors defined the 95th percentile error 

contours for the original error surface as well as the reprocessed error surface. The difference 

between the two 95th percentile contours was due to a correction of the 3-inch EFM calibrations 

when the exhaust flow rate error surface was reprocessed. SwRI and Sensors Inc. had

recalibrated the 3-inch flow meters using data generated at SwRI. The SwRI calibration

increased the slope multiplier by approximately 4 % for each 3-inch flow meter. Later in the 

program, Sensors Inc. discovered the 3-inch EFMs were likely not operating correctly on the

SwRI flow stand due to EFM pressures below ambient levels. Therefore, each 3-inch flow meter 

calibration was corrected to the original coefficients generated by Sensors. The corrected 3-inch 

flow meter data generated the 95th percentile contour shown in Figure 1. Because the 95th

percentile errors were larger than the 5th percentile errors, the 95th percentile data was mirrored to 

generate the 5th percentile contour in the reprocessed error surface. 

SwRI Report 03.12859.06 2 of 83 
-6

-4

-2

0

2

4

6

8

10

0 10 20 30 40 50 60 70 8

Lab Reference Mean Exhaust Flow Rate [% of Max]

PEMS Exhaust Flo

w Rate Delta [% of Max]

95th percentile 50th percentile (median) 5th percentile

Previous 95th Previous 50th Previous 5th

FIGURE 1. REVISED ERROR SURFACE FOR STEADY-STATE EXHAUST FLOW 

RATE 

2.2 Steady-State CO2 Error Surface 

Figure 2 shows the reprocessed, level-dependent CO2 error surface. The 95th percentile 

contour represents the envelope that was created to encompass the original 95th percentile error 

data. Line segments were used to connect the most extreme points from the original 95th

percentile data. The 95th percentile contour was mirrored to produce the 5th percentile error data, 

with the 50th percentile deltas set to zero. The reprocessed error surface was meant to encompass

all possible PEMS CO2 measurement errors, not just those measured at SwRI. 

0

SwRI Report 03.12859.06 3 of 83 
-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0

Lab Reference Mean CO2 Concentration [%]

PEMS CO2 Concentration Delta [%]

95th percentile 50th percentile (median) 5th percentile

Previous 95th Previous 50th Previous 5th

FIGURE 2. REVISED ERROR SURFACE FOR STEADY-STATE CO2

2.3 Steady-State CO Error Surface 

Figure 3 shows the revised steady-state error surface for CO. Because the CO error data 

showed no level dependence, a statistical calculation process, developed by Bill Martin from 

Cummins Inc., was used to generate the reprocessed CO error surface. The total standard 

deviation of the pooled CO error data was calculated by combining the variance of the PEMS 

mean delta standard deviation about zero and the PEMS pooled estimate of repeatability standard 

deviation. The calculation of the total standard deviation is shown below. 

SDindiv,pems = standard deviation of each PEMS calculated over 20 repeats of each steady-state 

point 

n = 10 steady-state points per engine * 3 engines * 3 PEMS = 90 

2

,

2

total repeatability mean zero SD = SD + SD

n

SD

SD

n

indiv pems

repeatability

∑ = 1

2

,

SwRI Report 03.12859.06 4 of 83 
PEMSindiv,mean = mean delta of each PEMS calculated over 20 repeats of each steady-state point 

Shown below, the multiplier for a 90% confidence interval assuming a normal 

distribution for the errors was applied to the total standard deviation (48.84 ppm = 0.004882%) 

to generate the constant 5th and 95th percentile error values. 

-0.010

-0.008

-0.006

-0.004

-0.002

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.0000 0.0005 0.0010 0.0015 0.0020 0.0025

Lab Reference Mean CO Concentration [%]

PEMS CO Concentration Delta [%]

95th percentile 50th percentile (median) 5th percentile

Previous 95th Previous 50th Previous 5th

For reprocessing the original stead

generate three NOx concentration error surfaces. For each of the three modified NOx error 

surfaces, the original set of Engine 2 steady-state data was added to the original pooled data set.

The Engine 2 steady-state data was collected in a repeat run in the original study due to PEMS 

failures that caused a large time gap in the original data. The time gap was associated with a 

NOx concentration shift. Since the NOx shift would have caused problems with the steady-state 

( )

1

0

1

2

,

, −

−

=

∑

n

PEMS

SD

n

indiv mean

mean zero

95 % 1.645* 0.008%

5 % 1.645* 0.008%

= = +

= − = −

total

th

total

th

SD

SD

FIGURE 3. REVISED ERROR SURFACE FOR STEADY-STATE CO 

2.4 Steady-State NOx Error Surface 

y-state NOx, three different methods were used to 

SwRI Report 03.12859.06 5 of 83 
variance correction of the transient data the repeat run data was used. However, the original 

Engine 2 delta data was later determined to be valid and thus was included in the modified data 

set to increase the PEMS delta observations. 

steady-state CO error surface. Using the complete NOx delta data set, the 5th and 95th percentile 

error values were calculated with the total standard deviation (15.08 ppm). The results of the 

first NOx error surface modification are shown in Figure 4 with the 5th and 95th percentiles set to 

±24.80 ppm. 

-100

-80

-60

-40

-20

0

20

40

100 200 300 400 500 600 700

Lab Reference Mean NOx Concentration [ppm]

PEMS N

Ox Concentration Delta [pp

m]

95th percentile 50th percentile (median) 5th percentile

Previous 95th Previous 50th Previous 5th

except six outlying low delta values recorded during Engine 3 testing were removed from the

pooled data set. During Engine 3 steady-state testing, PEMS 1 and 6 showed extremely low 

deltas at high NOx concentration levels. PEMS 4 also showed low biases at high NOx levels; 

however, the biases were not as large as those measured for the other PEMS. Therefore, PEMS

1 and 6 delta measurements at the three highest NOx levels measured during engine testing were 

removed from the data set. This resulted in a total standard deviation equal to 9.39 ppm and the 

5th and 95th percentile deltas equal to ±15.45 ppm. The results of this analysis are shown in 

Figure 5. 

The first NOx error surface modification (Mod 1) followed a procedure similar to the 

FIGURE 4. REVISED ERROR SURFACE MOD 1 FOR STEADY-STATE NOX

The second NOx error surface modification (Mod 2) was similar to the first modification 

SwRI Report 03.12859.06 6 of 83 
-100

-80

-60

-40

-20

0

20

40

100 200 300 400 500 600 700

Lab Reference Mean NOx Concentration [ppm]

PEMS N

Ox Concentration Delta [pp

m]

95th percentile 50th percentile (median) 5th percentile

Previous 95th Prefious 50th Previous 5th

FIGURE 5. REVISED ERROR SURFACE MOD 2 FOR STEADY-STATE NOX

 Shown in Figure 6, the third NOx error surface modification (Mod 3) combined the leveldependent and level-independent analysis methods. Below 311 ppm, the delta data was assumed 

to be level independent. Therefore, a statistical method similar to that used for CO was used for 

the NOx delta data below 311 ppm. The total standard deviation for the level-independent data 

was 9.41 ppm and the resulting 5th and 95th percentiles were ± 15.48 ppm. Above 311 ppm, the

original 5th percentile profile was used and mirrored to the 95th percentile to generate a 

symmetric error surface.

SwRI Report 03.12859.06 7 of 83 
-100

-80

-60

-40

-20

0

20

40

60

80

100

100 200 300 400 500 600 700

Lab Reference Mean NOx Concentration [ppm]

PEMS N

Ox Concentration Delta [pp

m]

95th percentile 50th percentile (median) 5th percentile

Previous 95th Previous 50th Previous 5th

FIGURE 6. REVISED ERROR SURFACE MOD 3 FOR STEADY-STATE NOX

In addition to running the MC simulations with the three different changes to the steadystate NOX error surface, it was useful to run a simulation with the original steady-state NOX error 

surface and only include the revised steady-state exhaust flow rate and steady-state CO2 error 

surfaces. This set of simulations is called the Mod B runs. Table 2 provides a summary of the 

four MC simulation runs made in this study with the revised error surfaces that were included in 

each run. All the other error surfaces that are not listed in Table 2 are the same as those run in 

the original PEMS study. 

TABLE 2. REVISED ERROR SURFACES USED IN MONTE CARLO SIMULATIONS 

Error Surface Mod B Run Mod 1 Run Mod 2 Run Mod 3 Run 

Steady-State Exhaust Flow 

Rate 

X X X X 

Steady-State CO2 X X X X 

Steady-State CO X X X 

Steady-State NOx Mod 1 X 

Steady-State NOx Mod 2 X 

Steady-State NOx Mod 3 X 

SwRI Report 03.12859.06 8 of 83 
3.0 REFERENCE NTE EVENTS 

The Monte Carlo simulation results from the original study included a reference data set 

consisting of 195 NTE events gathered from a number of sources. These included five engine 

manufacturers, SwRI transient lab tests and pre-pilot CE-CERT data. The investigations 

performed in this modified program included only a subset of the original 195 reference NTE 

events. Simulations from three different subsets of the original NTE events were run to study the 

validation of the emissions by the three calculation methods using the revised error surfaces. 

Table 3 lists the NTE events along with their ideal BSNOx values selected for each 

simulation. The various NTE subsets are described as follows: 

• 13 Reference Events – These are the same 13 NTE events chosen during the original 

study to investigate the error surface sensitivities due to bias and variance. These events 

were selected to bound the BSNOx threshold of 2.6820 g/kW-hr. These 13 NTE events 

were used to examine the error surface sensitivities for both the validation and full model 

simulation runs. 

• 23 Reference Events – Ten additional NTE events were added to the original 13 NTE 

events described above in order to increase the sample size for generating the validation 

EDF plots using the drift corrected CE-CERT data. 

• 50 Reference Events – After the model validation was confirmed for the three emissions 

across the three calculation methods, Mods 1 and 2 were chosen for continued analysis. 

Twenty-seven additional reference NTE events were simulated with the Monte Carlo 

model in order to estimate the measurement allowances for each emission. The selection

of the additional 27 reference NTE events was based on maintaining a similar distribution 

of ideal BSNOx values for Method 1 established from the original study containing 195 

reference NTE events. 

Figure 7 shows comparison histograms of the Method 1 BSNOx values for the 23 

reference NTE events (lower histogram) and the 195 reference NTE events (upper histogram)

while Figure 8 compares the Method 1 BSNOx histograms for the 50 reference NTE events 

(lower histogram) and the 195 reference NTE events (upper histogram). Descriptive statistics of 

BSNOx for these NTE subsets are detailed in Table 4. 

SwRI Report 03.12859.06 9 of 83 
TABLE 3. REFERENCE NTE EVENTS USED IN MONTE CARLO SIMULATIONS 

NTE # Ideal NOx (g/kW-hr) 

Method 1 

13 Reference 

Events 

23 Reference 

Events 

50 Reference 

Events 

1 4.0713 X X X 

3 3.0668 X X 

4 3.5832 X 

7 5.2516 X X 

11 1.8583 X 

16 2.3511 X 

20 3.7245 X X 

22 4.7916 X 

23 3.1483 X 

25 5.4061 X X X 

29 5.5261 X X 

37 5.4511 X 

38 0.0250 X X X 

40 4.1675 X 

43 1.1473 X 

44 1.0730 X X X 

46 2.6958 X X X 

51 2.8299 X X X 

57 4.3382 X X 

63 2.6670 X X X 

65 2.7437 X 

66 3.9378 X 

67 5.8600 X 

69 3.0257 X X X 

71 6.6867 X 

82 2.4569 X X X 

86 1.7132 X 

87 1.5207 X X X 

89 2.2566 X X 

92 1.6041 X 

96 1.6224 X X 

99 1.8147 X 

103 1.9186 X 

115 1.3854 X 

125 2.3214 X 

127 3.3005 X 

136 2.6782 X X 

139 2.4018 X X 

146 2.3053 X 

148 1.9985 X X X 

157 3.4666 X X X 

SwRI Report 03.12859.06 10 of 83 
160 2.0773 X 

162 2.1405 X 

163 2.5908 X X X 

168 1.5859 X 

176 1.9671 X X 

177 1.7507 X 

179 2.2023 X 

191 6.0815 X 

193 5.7521 X 

Ideal NOx Method 1 195 Ref NTE

Ideal NOx Method 1 23 Ref NTE

Fr

e

q

u

e

n

cy

0 2 4 6 8

20

10

0

10

20

30

40

FIGURE 7. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 23 

SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS 

SwRI Report 03.12859.06 11 of 83 
Ideal NOx Method 1 195 Ref NTE Events

Ideal NOx Method 1 50 Ref NTE Events Frequency

0 2 4 6 8

20

10

0

10

20

30

40

FIGURE 8. DISTRIBUTION OF IDEAL NOX (G/KW-HR) METHOD 1 FOR 50 

SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS 

TABLE 4. DESCRIPTIVE STATISTICS FOR IDEAL BSNOX (G/KW-HR) FOR 

VARIOUS NTE SUBSETS 

Descriptive Statistic 23 Reference NTE 

Events 

50 Reference NTE 

Events 

195 Reference NTE 

Events 

Minimum 0.0249 0.0249 0.0249 

Maximum 5.5261 6.6867 7.1927 

Mean 2.8983 3.0068 3.0071 

Median 2.6782 2.6289 2.6033 

Standard Deviation 1.3671 1.5175 1.4807 

SwRI Report 03.12859.06 12 of 83 
4.0 CE-CERT MEASUREMENTS 

The CE-CERT on-road measurements collected during the original PEMS study included 

delta BS emissions for all three emissions and three calculation methods. As part of this study to 

investigate all possible reasons why BSNOx did not validate for Methods 2 and 3, the 100 NTE

events from the CE-CERT data were examined for correctness by EPA. Since the CE-CERT 

data had not been drift corrected in the original PEMS study, EPA performed drift corrections on 

all three emissions for the 100 CE-CERT NTE events. As a result, several CE-CERT NTE 

events did not pass the drift check criteria and, therefore, were not included in the simulation

performed for this study. In addition, time alignment problems were found in a few of the 100 

CE-CERT NTE events and these were also excluded from the simulation. After deleting the 

NTE events due to drift check or time alignment problems, the on-road delta BS emissions were 

calculated from 81 NTE events for BSNOx and 87 NTE events for BSNMHC and BSCO. These 

remaining CE-CERT NTE events were all drift corrected. In contrast, the original PEMS 

program did not use drift correction in the CE-CERT on-road NTE events. 

SwRI Report 03.12859.06 13 of 83 
5.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE 

EVENTS USING MOD 1, MOD 2 AND MOD 3 

The Monte Carlo simulations performed using the 23 reference NTE events listed in

Table 3 included modifications for the error surfaces listed in Table 2. All the error surfaces

from the original MC simulations were used except those listed as revised in Table 2. Each of 

the 23 reference NTE events were run using the error surfaces revised for Mod 1 for either

10,000 or 30,000 trials. If the ideal BSNOX for an NTE event was less than 2.68204 g/kW-hr 

then the simulation was run for 10,000 trials. Otherwise, the NTE was simulated using 30,000 

trials. Once the MC simulations were completed using the Mod 1 revised error surfaces, the 

same 23 NTE events were simulated a second time with the error surface modifications for Mod 

2. Lastly, the 23 reference NTE events were simulated using the error surface modifications for 

Mod 3. 

In Mod 1, Mod 2, and Mod 3 simulation runs, all three emissions converged within 1% of 

the emissions threshold value by all three calculation methods. Table 5 lists the convergence 

criteria for all three brake-specific emissions. Thus, all 23 reference NTE events met the 

convergence criteria. 

TABLE 5. CONVERGENCE CRITERIA BY EMISSION 

Emission 1% of Threshold, g/kW-hr 

BSNOx 0.02682 

BSNMHC 0.00282 

BSCO 0.26015 

From the MC simulations, the validation 5th and 95th percentile delta BS emissions were 

extracted from the output files for each of the 23 reference NTE events. These delta emissions

were then plotted as empirical distribution functions (EDF) to form a validation interval for the

on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events. 

Figure 9 through Figure 11 represent the validation plots for the BSNOx for calculation methods 

1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the 

Mod 1, Mod 2, and Mod 3 runs. Figure 12 through Figure 17 depict the validation plots for the 

BSNMHC and BSCO simulation runs, respectively. The validation criteria set by the Steering 

Committee for the original study was used in the modified program. It included the following 

criteria: 

• At least 90% of the CE-CERT emissions deltas must be within the 5th and 95th

percentiles of the MC validation cumulative emissions deltas. 

• No more than 10% of the CE-CERT emissions deltas may fall less than the 5th

percentile or greater than the 95th percentile. This may indicate that the model is biased

low or high. 

• Validation must be shown for all three calculation methods. 

SwRI Report 03.12859.06 14 of 83 
A summary of the BSNOx validation conclusions based on the 23 reference NTE events 

is provided in Table 6. All BSNOX, BSNMHC and BSCO emissions validated for all three 

calculation methods and all three Mod runs. 

TABLE 6. BSNOX VALIDATION RESULTS BASED ON 23 REFERENCE NTE

EVENTS 

Simulation 

Run 

Calculation Method 

1 2 3 

Mod 1 

 · All CE-CERT data > 

5th percentile 

 · All CE-CERT data < 

95th percentile 

 · VALID 

 · All CE-CERT data > 

5th percentile 

 · 1% of CE-CERT data 

> 95th percentile 

 · VALID 

 · All CE-CERT data > 

5th percentile 

 · All CE-CERT data < 

95th percentile 

 · VALID 

Mod 2 

 · All CE-CERT data > 

5th percentile 

 · All CE-CERT data < 

95th percentile 

 · VALID 

 · All CE-CERT data > 

5th percentile 

 · 1% of CE-CERT data 

> 95th percentile 

 · VALID 

 · All CE-CERT data > 

5th percentile 

 · 1% of CE-CERT data 

> 95th percentile 

 · VALID 

Mod 3 

 · All CE-CERT data > 

5th percentile 

 · All CE-CERT data < 

95th percentile 

 · VALID 

 · All CE-CERT data > 

5th percentile 

 · All CE-CERT data < 

95th percentile 

 · VALID 

 · All CE-CERT data > 

5th percentile 

 · All CE-CERT data < 

95th percentile 

 · VALID 

SwRI Report 03.12859.06 15 of 83 
FIGURE 9. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NOX METHOD 1 MODS 1, 2 AND 3 

SwRI Report 03.12859.06 16 of 83 
FIGURE 10. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NOX METHOD 2 FOR MODS 1, 2 AND 3 

SwRI Report 03.12859.06 17 of 83 
FIGURE 11. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NOX METHOD 3 FOR MODS 1, 2 AND 3 

SwRI Report 03.12859.06 18 of 83 
FIGURE 12. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NMHC METHOD 1 FOR MODS 1, 2 AND 3

SwRI Report 03.12859.06 19 of 83 
FIGURE 13. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NMHC METHOD 2 FOR MODS 1, 2 AND 3

SwRI Report 03.12859.06 20 of 83 
FIGURE 14. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NMHC METHOD 3 FOR MODS 1, 2 AND 3

SwRI Report 03.12859.06 21 of 83 
FIGURE 15. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

CO METHOD 1 FOR MODS 1, 2 AND 3 

SwRI Report 03.12859.06 22 of 83 
FIGURE 16. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

CO METHOD 2 MODS 1, 2 AND 3 

SwRI Report 03.12859.06 23 of 83 
FIGURE 17. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

CO METHOD 3 FOR MODS 1, 2 AND 3 

SwRI Report 03.12859.06 24 of 83 
6.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE 

EVENTS USING MOD B 

Since all three methods validated using the 23 reference NTE events by all three 

modification runs for BSNOx, BSNMHC and BSCO, the question arose as to which of the

revised error surfaces had the most influence in the validation. Three possible explanations 

included (1) the steady-state CO2 bias had been eliminated, (2) the change in the steady-state 

NOX error values, and (3) the bias in the steady-state exhaust flow rate had been eliminated. To 

study these possible explanations, none of the revised SSNOx error surfaces or the SSCO error

surface was used in the simulations. These error surfaces were set to their original format in the 

original PEMS study. Therefore, only the steady-state exhaust flow rate and the steady-state

CO2 error surfaces were revised and the other remaining validation error surfaces were set to 

their original definitions for running the 23 reference NTE events for the Mod B MC simulation. 

This set of simulations is referred to as the ‘Mod B’ runs. Again, the simulations were 

performed at 10,000 trials for NTE events with ideal BSNOx values less than 2.68204 g/kW-hr 

and at 30,00 trials otherwise. All 23 reference NTE events met the convergence criteria listed in 

Table 5. 

From the Mod B MC simulations, the validation 5th and 95th percentile delta BS 

emissions were extracted from the output files for each of the 23 reference NTE events. These 

delta emissions were then plotted as empirical distribution functions (EDF) to form a validation 

interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT 

NTE events. Figure 18 through Figure 20 represent the validation plots for the BSNOx for 

calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th

percentile EDFs for the Mod B runs and the on-road CE-CERT data. Figure 21 through Figure 

26 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. All 

BSNOx, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod 

B runs. 

Conclusions made from the results of the Mod B analysis were that the steady-state NOX 

error surface changes did not have as much of an effect on the validation as the bias elimination

in the steady-state CO2 and steady-state exhaust flow rate error surfaces. 

SwRI Report 03.12859.06 25 of 83 
FIGURE 18. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NOX METHOD 1 MOD B 

SwRI Report 03.12859.06 26 of 83 
FIGURE 19. VALIDATION NTE EVENTS USING 23 REFERENCE NTE EVENTS FOR 

NOX METHOD 2 MOD B 

SwRI Report 03.12859.06 27 of 83 
FIGURE 20. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NOX METHOD 3 MOD B 

SwRI Report 03.12859.06 28 of 83 
FIGURE 21. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NMHC METHOD 1 MOD B 

SwRI Report 03.12859.06 29 of 83 
FIGURE 22. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NMHC METHOD 2 MOD B 

SwRI Report 03.12859.06 30 of 83 
FIGURE 23. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

NMHC METHOD 3 MOD B 

SwRI Report 03.12859.06 31 of 83 
FIGURE 24. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

CO METHOD 1 MOD B 

SwRI Report 03.12859.06 32 of 83 
FIGURE 25. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

CO METHOD 2 MOD B 

SwRI Report 03.12859.06 33 of 83 
FIGURE 26. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR 

CO METHOD 3 MOD B 

SwRI Report 03.12859.06 34 of 83 
7.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE 

EVENTS USING MOD 1 

Based on the information provided in the validation plots for the Mod 1, Mod 2, Mod 3 

and Mod B runs, changes made to the various steady-state error surfaces resulted in the 

validation of the MC model for all three emissions and all three calculation methods for each of 

the four Mod runs using the 23 reference NTE events. EPA chose to continue the simulation

runs using only the Mod 1 and Mod 2 revised error surfaces by running an additional 27 

reference NTE events (total = 50 reference NTE events) through the MC model. These two 

Mods were chosen because they both represented steady-state NOx error surfaces that were level 

independent (as compared to the Mod 3 steady-state error surface that represented a combination 

of level dependent and level independent NOX errors). The results from these 50 simulations 

were used to calculate the measurement allowances provided in Section 9.0 of this report. This

section details the results of the validation based on the 50 reference NTE events run with Mod 

1. 

The Mod 1 simulations were performed at 10,000 trials for NTE events with ideal 

BSNOX values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 50 reference NTE

events met the convergence criteria listed in Table 5. 

From the Mod 1 MC simulations, the validation 5th and 95th percentile delta BS emissions 

were extracted from the output files for each of the 50 reference NTE events. These percentiles 

were then plotted as empirical distribution functions (EDF) to form a validation interval for the

on-road data. Also plotted was the EDF computed from the on-road CE-CERT NTE events. 

Figure 27 through Figure 29 represent the validation plots for the BSNOx for calculation methods 

1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th percentile EDFs for the 

Mod 1 runs and the on-road CE-CERT data. Figure 30 through Figure 35 depict the validation 

plots for the BSNMHC and BSCO simulation runs, respectively. All BSNOX, BSNMHC and 

BSCO emissions validated for all three calculation methods for the Mod 1 runs with the 50 

reference NTE events. 

SwRI Report 03.12859.06 35 of 83 
FIGURE 27. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NOX METHOD 1 MOD 1 

SwRI Report 03.12859.06 36 of 83 
FIGURE 28. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NOX METHOD 2 MOD 1 

SwRI Report 03.12859.06 37 of 83 
FIGURE 29. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NOX METHOD 3 MOD 1 

SwRI Report 03.12859.06 38 of 83 
FIGURE 30. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NMHC METHOD 1 MOD 1 

SwRI Report 03.12859.06 39 of 83 
FIGURE 31. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NMHC METHOD 2 MOD 1 

SwRI Report 03.12859.06 40 of 83 
FIGURE 32. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NMHC METHOD 3 MOD 1 

SwRI Report 03.12859.06 41 of 83 
FIGURE 33. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

CO METHOD 1 MOD 1 

SwRI Report 03.12859.06 42 of 83 
FIGURE 34. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

CO METHOD 2 MOD 1 

SwRI Report 03.12859.06 43 of 83 
FIGURE 35. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

CO METHOD 3 MOD 1 

SwRI Report 03.12859.06 44 of 83 
8.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE 

EVENTS USING MOD 2 

This section details the results of the validation based on the 50 reference NTE events run 

with Mod 2. EPA chose to continue simulations using the Mod 1 and Mod 2 revised error 

surfaces by running an additional 27 reference NTE events (total = 50 reference NTE events)

through the MC model. These two Mods were chosen because they both represented steadystate NOx error surfaces that were level independent (as compared to the Mod 3 steady-state 

error surface that represented a combination of level dependent and level independent NOX

errors). 

The Mod 2 simulations were performed at 10,000 trials for NTE events with ideal 

BSNOX values less than 2.68204 g/kW-hr and at 30,00 trials otherwise. All 50 reference NTE

events met the convergence criteria listed in Table 5. 

From the Mod 2 MC simulations, the validation 5th and 95th percentile delta BS emissions 

were extracted from the output files for each of the 50 reference NTE events. These percentile 

delta emissions were then plotted as empirical distribution functions (EDF) to form a validation 

interval for the on-road data. Also plotted was the EDF computed from the on-road CE-CERT 

NTE events. Figure 36 through Figure 38 represent the validation plots for the BSNOx for 

calculation methods 1, 2 and 3, respectively. Each of the validation plots includes 5th and 95th

percentile EDFs for the Mod 2 runs and the on-road CE-CERT data. Figure 39 through Figure 

44 depict the validation plots for the BSNMHC and BSCO simulation runs, respectively. All 

BSNOX, BSNMHC and BSCO emissions validated for all three calculation methods for the Mod 

2 runs with the 50 reference NTE events. 

SwRI Report 03.12859.06 45 of 83 
FIGURE 36. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NOX METHOD 1 MOD 2 

SwRI Report 03.12859.06 46 of 83 
FIGURE 37. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NOX METHOD 2 MOD 2 

SwRI Report 03.12859.06 47 of 83 
FIGURE 38. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR 

NOX METHOD 3 MOD 2 

SwRI Report 03.12859.06 48 of 83 
FIGURE 39. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR 

NMHC METHOD 1 MOD 2 

SwRI Report 03.12859.06 49 of 83 
FIGURE 40. VALIDATION EDF PLOTS FOR 50 REFERENCE NTE EVENTS FOR 

NMHC METHOD 2 MOD 2 

SwRI Report 03.12859.06 50 of 83 
FIGURE 41. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

NMHC METHOD 3 MOD 2 

SwRI Report 03.12859.06 51 of 83 
FIGURE 42. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

CO METHOD 1 MOD 2 

SwRI Report 03.12859.06 52 of 83 
FIGURE 43. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

CO METHOD 2 MOD 2 

SwRI Report 03.12859.06 53 of 83 
FIGURE 44. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR 

CO METHOD 3 MOD 2 

Table 7 provides a summary of all the validation results for the various MC simulations

as described in the above sections of this report. All emissions and all calculation methods

validated using the Mod 1 and Mod 2 runs with 50 reference NTE events while the Mod 3 and 

Mod B runs validated using 23 reference NTE events. 

SwRI Report 03.12859.06 54 of 83 
TABLE 7. SUMMARY OF VALIDATION RESULTS 

Emission Original

195 NTE 

Mod 1 

50 NTE 

Mod 2 

50 NTE 

Mod 3 

23 NTE 

Mod B 

23 NTE 

BSNOX

Method 1 Validated Validated Validated Validated Validated 

BSNOX

Method 2 Not Validated Validated Validated Validated Validated 

BSNOX

Method 3 Not Validated Validated Validated Validated Validated 

BSNMHC 

Method 1 Validated Validated Validated Validated Validated 

BSNMHC 

Method 2 Validated Validated Validated Validated Validated 

BSNMHC 

Method 3 Validated Validated Validated Validated Validated 

BSCO 

Method 1 Not Validated Validated Validated Validated Validated 

BSCO 

Method 2 Not Validated Validated Validated Validated Validated 

BSCO 

Method 3 Not Validated Validated Validated Validated Validated 

SwRI Report 03.12859.06 55 of 83 
9.0 MEASUREMENT ALLOWANCE CALCULATIONS 

As detailed in the original PEMS study, the measurement error allowances were 

computed using both a regression method and a median method to determine the measurement 

allowance. The procedure was applied to the simulation data for the 50 reference NTE events

from the Mod 1 and Mod 2 runs for each of the three emissions and all three calculation 

methods. 

Figure 45 contains a regression plot of the 95th percentile delta BSNOx values (using 

Method 1 Mod 1) versus the ideal BSNOx values for the 50 reference NTE events. Included 

within the plot is the equation for the fitted regression line and the R-square (R2

) value. Table 8

includes a comparison of the results of the regression method based on Figure 45 and the median 

method (described in Ref 1). Under the heading “Regression Method” in the table, it is shown 

that the R-square criterion is not met by the data (R-square must be > 0.90). Thus, the median 

method must be used. Under the heading “Median Method” in the table, the measurement error 

at the BSNOx threshold, based on using the median of the fifty 95th percentile delta BSNOx

values, is 11.5932% when expressed as a percent of the threshold value of 2.68204 g/kW-hr. 

Similar regression plots and measurement error tables are provided in the remaining part 

of this section for the Mod 1 and Mod 2 results based on the 50 reference NTE events. Figure 45

through Figure 47 and Table 8 through Table 10 provide results for BSNOx Mod 1. Figure 48

through Figure 50 and Table 11 through Table 13 provide results for BSNMHC Mod 1, and 

Figure 51 through Figure 53 and Table 14 through Table 16 provide results for BSCO Mod 1. 

Mod 2 results for BSNOx can be found in Figure 54 through Figure 56 and Table 17 through 

Table 19, BSNMHC results are shown in Figure 57 through Figure 59 and Table 20 through 

Table 22 and BSCO results are shown in Figure 60 through Figure 62 and Table 23 through 

Table 25. 

SwRI Report 03.12859.06 56 of 83 
NOx g/kW-hr Method 1 Mod 1

50 Ref NTE Events With Time Alignment Adjustment

y = 0.1213x + 0.0387

R2 = 0.8487

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0 1 2 3 4 5 6 7

Ideal NOx g/kW-hr

95th % Delta N

Ox g/kW-hr

FIGURE 45. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS 

IDEAL BSNOX FOR METHOD 1 MOD 1 

TABLE 8. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 1 

Regression Method Median Method 

R2 0.8487 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0785 Met 

Criteria 

5% Median Ideal 0.1314 

Predicted 95th% Delta at 

Threshold 0.3640 Median 95th% Delta 0.3109 

Measurement Error @ 

Threshold=2.68204 13.5729% Measurement Error @ Threshold=2.68204 11.5932% 

8

SwRI Report 03.12859.06 57 of 83 
NOx g/kW-hr Method 2 Mod 1

50 Ref NTE Events With Time Alignment Adjustment

y = 0.0558x + 0.0893

R2 = 0.9058

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0 1 2 3 4 5 6 7 8

Ideal NOx g/kW-hr

95th % Delta NOx g/kW-hr

FIGURE 46. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS 

IDEAL BSNOX FOR METHOD 2 MOD 1 

TABLE 9. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1 

Regression Method Median Method 

R2 0.9058 Met 

Criteria 

RMSE(SEE) 0.0276 Met 

Criteria 

5% Median Ideal 0.1314 

Predicted 95th% Delta at 

Threshold 0.2390 Median 95th% Delta 0.2270 

Measurement Error @ 

Threshold=2.68204 8.9096% Measurement Error @ 

Threshold=2.68204 8.4641% 

SwRI Report 03.12859.06 58 of 83 
NOx g/kW-hr Method 3 Mod 1

50 Ref NTE Events With Time Alignment Adjustment

y = 0.0869x + 0.0533

R2 = 0.8714

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0 1 2 3 4 5 6 7 8

Ideal NOx g/kW-hr

95th % Delta N

Ox g/kW-hr

FIGURE 47. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS 

IDEAL BSNOX FOR METHOD 3 MOD 1 

TABLE 10. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1 

Regression Method Median Method 

R2 0.8714 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0512 Met 

Criteria 

5% Median Ideal 0.1314 

Predicted 95th% Delta at 

Threshold 0.3000 Median 95th% Delta 0.2572 

Measurement Error @ 

Threshold=2.68204 11.1845% Measurement Error @ Threshold=2.68204 9.5910% 

SwRI Report 03.12859.06 59 of 83 
NMHC g/kW-hr Method 1 Mod 1

50 Ref NTE Events

y = 0.0955x + 0.0174

R2 = 0.8109

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45

Ideal NMHC g/kW-hr

95th % Delta NMHC g/kW-hr

FIGURE 48. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC 

VERSUS IDEAL BSNMHC FOR METHOD 1 MOD 1 

TABLE 11. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 1 

Regression Method Median Method 

R2 0.8109 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0037 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0002 

Predicted 95th% Delta at 

Threshold 0.0443 Median 95th% Delta 0.0197 

Measurement Error @ 

Threshold=0.28161 15.7288% Measurement Error @ Threshold=0.28161 7.0022% 

SwRI Report 03.12859.06 60 of 83 
NMHC g/kW-hr Method 2 Mod 1

50 Ref NTE Events

y = 0.0382x + 0.0174

R2 = 0.4768

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

0.045

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45

Ideal NMHC g/kW-hr

95th % Delta NMHC g/kW-hr

FIGURE 49. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC 

VERSUS IDEAL BSNMHC FOR METHOD 2 MOD 1 

TABLE 12. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1 

Regression Method Median Method 

R2 0.4768 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0032 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0002 

Predicted 95th% Delta at 

Threshold 0.0282 Median 95th% Delta 0.0187 

Measurement Error @ 

Threshold=0.28161 9.9988% Measurement Error @ Threshold=0.28161 6.6274% 

SwRI Report 03.12859.06 61 of 83 
NMHC g/kW-hr Method 3 Mod 1

50 Ref NTE Events

y = 0.0545x + 0.0171

R2 = 0.6202

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

0.045

0.050

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45

Ideal NMHC g/kW-hr

95th % Delta NMHC g/kW-hr

FIGURE 50. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC 

VERSUS IDEAL BSNMHC FOR METHOD 3 MOD 1 

TABLE 13. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1 

Regression Method Median Method 

R2 0.6202 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0034 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0002 

Predicted 95th% Delta at 

Threshold 0.0324 Median 95th% Delta 0.0188 

Measurement Error @ 

Threshold=0.28161 11.5222% Measurement Error @ Threshold=0.28161 6.6826% 

SwRI Report 03.12859.06 62 of 83 
CO g/kW-hr Method 1 Mod 1

50 Ref NTE Events With Time Alignment Adjustment

y = 0.1112x + 0.3148

R2 = 0.6518

0.00

0.20

0.40

0.60

0.80

1.00

1.20

0 1 2 3 4 5 6 7

Ideal CO g/kW-hr

95th % Delta C

O g/kW-hr

FIGURE 51. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS 

IDEAL BSCO FOR METHOD 1 MOD 1 

TABLE 14. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 1 

Regression Method Median Method 

R2 0.6518 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0846 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0249 

Predicted 95th% Delta at 

Threshold 0.9814 Median 95th% Delta 0.3600 

Measurement Error @ 

Threshold=26.015 3.7726% Measurement Error @ Threshold=26.015 1.3840% 

SwRI Report 03.12859.06 63 of 83 
CO g/kW-hr Method 2 Mod 1

50 Ref NTE Events With Time Alignment Adjustment

y = 0.0543x + 0.3211

R2 = 0.3471

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0 1 2 3 4 5 6 7

Ideal CO g/kW-hr

95th % Delta C

O g/kW-hr

FIGURE 52. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS 

IDEAL BSCO FOR METHOD 2 MOD 1 

TABLE 15. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 1 

Regression Method Median Method 

R2 0.3471 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0775 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0249 

Predicted 95th% Delta at 

Threshold 0.6466 Median 95th% Delta 0.3451 

Measurement Error @ 

Threshold=26.015 2.4856% Measurement Error @ Threshold=26.015 1.3265% 

SwRI Report 03.12859.06 64 of 83 
CO g/kW-hr Method 3 Mod 1

50 Ref NTE Events With Time Alignment Adjustment

y = 0.0988x + 0.311

R2 = 0.5987

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

1.10

0 1 2 3 4 5 6 7

Ideal CO g/kW-hr

95th % Delta C

O g/kW-hr

FIGURE 53. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS 

IDEAL BSCO FOR METHOD 3 MOD 1 

TABLE 16. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 1 

Regression Method Median Method 

R2 0.5987 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0846 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0249 

Predicted 95th% Delta at 

Threshold 0.9033 Median 95th% Delta 0.3621 

Measurement Error @ 

Threshold=26.015 3.4722% Measurement Error @ Threshold=26.015 1.3919% 

SwRI Report 03.12859.06 65 of 83 
NOx g/kW-hr Method 1 Mod 2

50 Ref NTE Events With Time Alignment Adjustment

y = 0.1272x - 0.0086

R2 = 0.8638

-0.10

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0 1 2 3 4 5 6 7

Ideal NOx g/kW-hr

95th % Delta N

Ox g/kW-hr

FIGURE 54. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS 

IDEAL BSNOX FOR METHOD 1 MOD 2 

TABLE 17. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2 

Regression Method Median Method 

R2 0.8638 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0774 Met 

Criteria 

5% Median Ideal 0.1314 

Predicted 95th% Delta at 

Threshold 0.3326 Median 95th% Delta 0.2834 

Measurement Error @ 

Threshold=2.68204 12.3993% Measurement Error @ Threshold=2.68204 10.5681% 

8

SwRI Report 03.12859.06 66 of 83 
NOx g/kW-hr Method 2 Mod 2

50 Ref NTE Events With Time Alignment Adjustment

y = 0.0614x + 0.0375

R2 = 0.9543

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0 1 2 3 4 5 6 7

Ideal NOx g/kW-hr

95th % Delta N

Ox g/kW-hr

FIGURE 55. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS 

IDEAL BSNOX FOR METHOD 2 MOD 2 

TABLE 18. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2 

Regression Method Median Method 

R2 0.9543 Met 

Criteria 

RMSE(SEE) 0.0206 Met 

Criteria 

5% Median Ideal 0.1314 

Predicted 95th% Delta at 

Threshold 0.2022 Median 95th% Delta 0.1935 

Measurement Error @ 

Threshold=2.68204 7.5382% Measurement Error @ 

Threshold=2.68204 7.2141% 

8

SwRI Report 03.12859.06 67 of 83 
NOx g/kW-hr Method 3 Mod 2

50 Ref NTE Events With Time Alignment Adjustment

y = 0.0932x + 0.0056

R2 = 0.8998

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0 1 2 3 4 5 6 7

Ideal NOx g/kW-hr

95th % Delta N

Ox g/kW-hr

FIGURE 56. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNOX VERSUS 

IDEAL BSNOX FOR METHOD 3 MOD 2 

TABLE 19. MEASUREMENT ERROR AT THRESHOLD FOR BSNOX USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2 

Regression Method Median Method 

R2 0.8998 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0477 Met 

Criteria 

5% Median Ideal 0.1314 

Predicted 95th% Delta at 

Threshold 0.2556 Median 95th% Delta 0.2295 

Measurement Error @ 

Threshold=2.68204 9.5288% Measurement Error @ Threshold=2.68204 8.5551% 

8

SwRI Report 03.12859.06 68 of 83 
NMHC g/kW-hr Method 1 Mod 2

50 ref NTE Events

y = 0.0951x + 0.0173

R2 = 0.816

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45

Ideal NMHC g/kW-hr

95th % Delta NMHC g/kW-hr

FIGURE 57. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC 

VERSUS IDEAL BSNMHC FOR METHOD 1 MOD 2 

TABLE 20. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2 

Regression Method Median Method 

R2 0.8160 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0036 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0002 

Predicted 95th% Delta at 

Threshold 0.0441 Median 95th% Delta 0.0194 

Measurement Error @ 

Threshold=0.28161 15.6533% Measurement Error @ Threshold=0.28161 6.8769% 

SwRI Report 03.12859.06 69 of 83 
NMHC g/kW-hr Method 2 Mod 2

50 Ref NTE Events

y = 0.0386x + 0.0172

R2 = 0.479

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

0.045

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45

Ideal NMHC g/kW-hr

95th % Delta NMHC g/kW-hr

FIGURE 58. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC 

VERSUS IDEAL BSNMHC FOR METHOD 2 MOD 2 

TABLE 21. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2 

Regression Method Median Method 

R2 0.4790 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0032 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0002 

Predicted 95th% Delta at 

Threshold 0.0281 Median 95th% Delta 0.0182 

Measurement Error @ 

Threshold=0.28161 12.3993% Measurement Error @ Threshold=0.28161 6.4499% 

SwRI Report 03.12859.06 70 of 83 
NMHC g/kW-hr Method 3 Mod 2

50 Ref NTE Events

y = 0.054x + 0.017

R2 = 0.6195

0.010

0.015

0.020

0.025

0.030

0.035

0.040

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45

Ideal NMHC g/kW-hr

95th % Delta NMHC g/kW-hr

FIGURE 59. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC 

VERSUS IDEAL BSNMHC FOR METHOD 3 MOD 2 

TABLE 22. MEASUREMENT ERROR AT THRESHOLD FOR BSNMHC USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2 

Regression Method Median Method 

R2 0.6195 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0034 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0002 

Predicted 95th% Delta at 

Threshold 0.0322 Median 95th% Delta 0.0182 

Measurement Error @ 

Threshold=0.28161 11.4367% Measurement Error @ Threshold=0.28161 6.4801% 

SwRI Report 03.12859.06 71 of 83 
CO g/kW-hr Method 1 Mod 2

50 Ref NTE Events With Time Alignment Adjustment

y = 0.1156x + 0.3087

R2 = 0.6865

0.00

0.20

0.40

0.60

0.80

1.00

1.20

0 1 2 3 4 5 6 7

Ideal CO g/kW-hr

95th % Delta C

O g/kW-hr

FIGURE 60. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS 

IDEAL BSCO FOR METHOD 1 MOD 2 

TABLE 23. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 1 MOD 2 

Regression Method Median Method 

R2 0.6865 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0813 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0249 

Predicted 95th% Delta at 

Threshold 1.0017 Median 95th% Delta 0.3576 

Measurement Error @ 

Threshold=26.015 3.8505% Measurement Error @ Threshold=26.015 1.3746% 

SwRI Report 03.12859.06 72 of 83 
CO g/kW-hr Method 2 Mod 2

50 Ref NTE Events With Time Alignment Adjustment

y = 0.0567x + 0.3162

R2 = 0.3785

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0 1 2 3 4 5 6 7

Ideal CO g/kW-hr

95th % Delta C

O g/kW-hr

FIGURE 61. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS 

IDEAL BSCO FOR METHOD 2 MOD 2 

TABLE 24. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 2 MOD 2 

Regression Method Median Method 

R2 0.3785 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0756 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0249 

Predicted 95th% Delta at 

Threshold 0.6561 Median 95th% Delta 0.3361 

Measurement Error @ 

Threshold=26.015 2.5221% Measurement Error @ Threshold=26.015 1.2919% 

SwRI Report 03.12859.06 73 of 83 
CO g/kW-hr Method 3 Mod 2

50 Ref NTE Events With Time Alignment Adjustment

y = 0.1009x + 0.3065

R2 = 0.6195

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

1.10

0 1 2 3 4 5 6 7

Ideal CO g/kW-hr

95th % Delta C

O g/kW-hr

FIGURE 62. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSCO VERSUS 

IDEAL BSCO FOR METHOD 3 MOD 2 

TABLE 25. MEASUREMENT ERROR AT THRESHOLD FOR BSCO USING 

REGRESSION AND MEDIAN METHODS FOR METHOD 3 MOD 2 

Regression Method Median Method 

R2 0.6195 

Did Not 

Meet 

Criteria 

RMSE(SEE) 0.0827 

Did Not 

Meet 

Criteria 

5% Median Ideal 0.0249 

Predicted 95th% Delta at 

Threshold 0.9114 Median 95th% Delta 0.3513 

Measurement Error @ 

Threshold=26.015 3.5033% Measurement Error @ Threshold=26.015 1.3505% 

SwRI Report 03.12859.06 74 of 83 
Although only 23 reference NTE events were simulated under the Mod 3 and Mod B 

runs, the measurement errors were computed for all three emissions and calculation methods 

using the same regression and median methods. Table 26 lists a summary of the measurement 

errors computed from simulation results in this study along with the results from the original 

PEMS study. Note that BSNOx for methods 2 and 3 and BSCO for all three methods did not 

validate in the original study. 

TABLE 26. SUMMARY OF MEASUREMENT ERRORS AT RESPECTIVE 

THRESHOLD (%) 

Emission # Ref NTE 

Events Model Run Calculation Method 

Method 1 Method 2 Method 3 

BSNOX 195 Original 22.30 4.45* 6.61* 

50 Mod 1 11.59 8.91 9.59 

50 Mod 2 10.56 7.54 8.56 

23 Mod 3 11.78 8.01 10.78 

23 Mod B 11.93 8.26 10.33 

BSNMHC 195 Original 10.08 8.03 8.44 

50 Mod 1 7.00 6.63 6.68 

50 Mod 2 6.88 6.45 6.48 

23 Mod 3 7.28 6.90 6.87 

23 Mod B 7.25 6.82 6.78 

BSCO 195 Original 2.58* 1.99* 2.11* 

50 Mod 1 1.38 1.33 1.39 

50 Mod 2 1.37 1.29 1.35 

23 Mod 3 1.52 1.41 1.48 

23 Mod B 2.19 1.99 2.16 

*

 Methods did not validate 

The final BS measurement allowances were computed by multiplying each of the three

measurement errors (in percent) times their corresponding threshold values. Table 27 lists the

measurement allowances for all simulation Mod runs and the original PEMS study. The 

emissions thresholds (g/hp-hr) are provided in the first column of Table 27. 

SwRI Report 03.12859.06 75 of 83 
TABLE 27. SUMMARY OF MEASUREMENT ALLOWANCE, G/HP-HR 

Emission # Ref NTE 

Events Model Run Calculation Method 

Method 1 Method 2 Method 3 

BSNOx 

Threshold=2.0 

195 Original 0.446 0.089* 0.132* 

50 Mod 1 0.232 0.178 0.192 

50 Mod 2 0.211 0.151 0.171 

23 Mod 3 0.236 0.160 0.216 

23 Mod B 0.239 0.165 0.207 

BSNMHC 

Threshold=0.21 

195 Original 0.021 0.017 0.018 

50 Mod 1 0.015 0.014 0.014 

50 Mod 2 0.014 0.014 0.014 

23 Mod 3 0.015 0.014 0.014 

23 Mod B 0.015 0.014 0.014 

BSCO 

Threshold=19.40 

195 Original 0.501* 0.386* 0.409* 

50 Mod 1 0.268 0.258 0.270 

50 Mod 2 0.266 0.250 0.262 

23 Mod 3 0.295 0.274 0.287 

23 Mod B 0.425 0.386 0.419 

*

 Methods did not validate 

SwRI Report 03.12859.06 76 of 83 
10.0 VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS 

Simulation results for Mod 1, Mod 2, Mod 3 and Mod B from the 13 reference NTE 

events chosen in the original study produced sensitivity values from the validation runs due to

variance and bias for all three 95th percentile delta emissions by all three calculation methods. 

The tables below summarize the error surfaces in which either the contribution-to-variance 

normalized sensitivity average value or the ‘on/off’ bias check for the error surface was at least 

5% in absolute magnitude compared to all the other error surfaces. If the label in the error 

surface contains the words ‘OnOff’ then it represents a check for bias; otherwise, the error 

surface indicates a check for variance. If at least 5 of the 13 reference NTE events resulted in a

sensitivity average value > 5% or < -5% then the average contribution to variance is included in 

the tables. 

Table 28 lists the BSNOX sensitivity error surfaces associated with the original PEMS 

study for Mod B, Mod 1, Mod 2 and the Mod 3 Monte Carlo validation simulations performed 

during this project. Note in the output results that the bias due to steady-state exhaust flow rate

in Method 1 in the original study has been eliminated in the four modification simulations. In

addition, the steady-state CO2 bias has been eliminated, but a sensitivity component due to 

variance has been added for Methods 2 and 3. 

Table 29 lists the sensitivities due to BSNMHC. Note that no changes have occurred for 

the four modifications relative to the original study. This is due to the fact that the revised error 

surfaces did not effect the NMHC calculations. Table 30 summarizes the results of the BSCO 

sensitivities wherein the steady-state CO bias found in the original modeling has been eliminated 

in Mod 1, Mod 2 and Mod 3 and replaced by an increased sensitivity due to variance. 

SwRI Report 03.12859.06 77 of 83 
TABLE 28. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS 

ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSNOX

Method # Error Surface 

Avg Contribution to Normalized Variance, % 

Mod 

B 

Mod 

1 

Mod 

2 

Mod 

3 

1 

1 SSNOx 29.0 33.2 42.9 32.0 36.3 

20 SS Exhaust Flow 9.6 19.2 16.6 20.4 18.8 

51 NOx Time

Alignment 

9.0 10.7 9.1 10.9 10.6 

Pulse_Flow_OnOff 25.8 30.5 26.3 33.1 29.1 

SS Exhaust 

Flow_OnOff 

20.4 

2 

1 SSNOx 37.0 46.9 57.8 44.6 52.5 

2 TRNOx 5.5 6.6 

45 SSCO2 -40.1 -33.7 -43.6 -37.7 

51 NOx Time

Alignment 

6.7 7.5 7.6 6.2 

SSCO2_OnOff -54.3 

3 

1 SSNOx 35.6 44.0 54.9 41.6 49.4 

45 SSCO2 -37.0 -31.3 -39.6 -34.8 

51 NOx Time

Alignment 

10.4 13.9 11.5 15.8 12.9 

SSCO2_OnOff -50.9 

DEFINITIONS 

Original April 2007 Study: All original error surfaces 

Mod B Revised SS Exhaust Flow and SS CO2 error surfaces 

Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Original

SwRI Report 03.12859.06 78 of 83 
TABLE 29. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS 

ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSNMHC 

Method # Error Surface 

Avg Contribution to Normalized Variance, % 

Mod 

B 

Mod 

1 

Mod 

2 

Mod 

3 

1 

13 SSNMHC 7.1 7.1 7.0 6.9 6.6 

19 NMHC Ambient 60.9 62.8 63.2 62.5 63.2 

SS NMHC_OnOff 14.5 14.3 13.5 12.9 14.2 

2 

13 SSNMHC 6.7 6.8 6.8 6.9 6.4 

19 NMHC Ambient 64.5 66.6 66.7 66.3 66.9 

SS NMHC_OnOff 14.3 12.9 13.5 13.2 12.8 

3 

13 SSNMHC 6.7 6.9 6.8 7.0 6.4 

19 NMHC Ambient 64.4 66.5 66.6 66.1 66.8 

SS NMHC_OnOff 14.4 13.0 13.6 12.5 13.0 

DEFINITIONS 

Original April 2007 Study: All original error surfaces 

Mod B Revised SS Exhaust Flow and SS CO2 error surfaces 

Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Original

SwRI Report 03.12859.06 79 of 83 
TABLE 30. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS 

ACROSS 5 MONTE CARLO VALIDATION SIMULATIONS FOR BSCO 

Method # Error Surface 

Avg Contribution to Normalized Variance, % 

Mod 

B 

Mod 

1 

Mod 

2 

Mod 

3 

1 

7 SSCO 6.2 6.5 79.3 79.1 79.3 

52 CO Time

Alignment 

10.3 11.0 10.1 10.7 

SSCO_OnOff 82.0 82.2 

2 

7 SSCO 6.4 6.7 82.9 83.0 83.0 

52 CO Time

Alignment 

10.5 11.8 10.4 

SSCO_OnOff 83.6 84.6 

3 

7 SSCO 6.2 6.3 77.9 78.0 77.9 

52 CO Time

Alignment 

11.8 10.1 19.2 17.9 19.0 

SSCO_OnOff 80.8 81.0 

DEFINITIONS 

Original April 2007 Study: All original error surfaces 

Mod B Revised SS Exhaust Flow and SS CO2 error surfaces 

Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error 

surfaces 

Original

SwRI Report 03.12859.06 80 of 83 
11.0 FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS 

This section contains a summary of the error surfaces that contributed the most to bias 

and variance of the BSNOX emissions based on the full MC model. Simulation results from the 

13 reference NTE events produced sensitivity values for all three 95th percentile delta emissions 

by all three calculation methods. Table 31 lists the error surfaces in which either the

contribution-to-variance normalized sensitivity average value or the ‘on/off’ bias check for the

error surface was at least 5% in absolute magnitude compared to all the other error surfaces for 

BSNOX. If at least 5 of the 13 reference NTE events resulted in a sensitivity average value > 5%

or < -5% then the average contribution to variance is included in the table. The sensitivity results 

from the original study are included for comparison. 

The results from the BSNOX Method 1 sensitivities show that the steady-state exhaust 

flow rate bias from the original study has been eliminated in Mod B, Mod 1, Mod 2, and Mod 3. 

There is also a slight increase (from the original study) in the average sensitivity due to variation

in the steady-state exhaust flow rate with all the modification runs. Slight increases in the 

average sensitivity for the steady-state NOX error surface are seen in all modification runs except 

Mod 2. 

Method 2 results for BSNOX show that the steady-state CO2 bias effect and the NOX time 

alignment variation effect were eliminated in all the modification runs. However, new error

surfaces that showed sensitivities in the modification runs were the steady-state CO2 variation

effect and the BSFC DOE Test bias effect. There were also slight increases in the average 

sensitivity due to variation for the steady-state NOX error surface in all the modification runs. 

The steady-state CO2 bias effect for Method 3 was eliminated in all the modification runs 

and a variation effect due to steady-state CO2 was included. Also, there was a slight increase in 

the average sensitivity due to variation for the steady-state NOX error surface in all the

modification runs. 

SwRI Report 03.12859.06 81 of 83 
TABLE 31. SENSITIVITY RESULTS COMPARING 13 REFERENCE NTE EVENTS 

ACROSS 5 MONTE CARLO FULL MODEL SIMULATIONS FOR BSNOX

Method # Error Surface 

Avg Contribution to Normalized Variance, % 

Mod B Mod 

1 

Mod 

2 

Mod 

3 

1 

1 SSNOx 21.0 22.3 31.2 20.8 25.4 

20 SS Exhaust Flow 6.3 10.2 9.5 11.0 10.4 

31 Warm-up torque -23.2 -23.8 -20.7 -23.7 -22.6 

35 Engine Manuf 

Torque 

-9.8 -9.9 -8.7 -9.9 -9.4 

51 NOx Time

Alignment 

6.1 5.5 5.0 5.5 5.6 

Pulse_Flow_OnOff 14.7 15.1 13.8 16.4 14.8 

SS Exhaust 

Flow_OnOff 

14.6 

2 

1 SSNOx 29.5 33.3 43.9 30.4 37.1 

38 Warm-up BSFC 6.3 5.9 5.2 6.2 5.6 

42 Engine Manuf BSFC 13.6 14.1 12.3 15.8 14.0 

45 SSCO2 -23.2 -20.0 -24.1 -21.9 

51 NOx Time

Alignment 

6.1 

BSFC DOE 

Test_OnOff 

-5.9 -5.3 -6.5 -6.1 

SSCO2_OnOff -34.9 

3 

1 SSNOx 23.6 25.6 35.1 34.2 29.2 

31 Warm-up Torque -27.8 -28.5 -25.0 -29.4 -27.0 

35 Engine Manuf 

Torque 

-11.7 -11.9 -10.4 -12.6 -11.8 

45 SSCO2 -16.3 -14.8 -16.8 -15.6 

51 NOx Time

Alignment 

6.0 5.4 4.8 5.8 5.3 

SSCO2_OnOff -25.2 

DEFINITIONS 

Original April 2007 Study: All original error surfaces 

Mod B Revised SS Exhaust Flow and SS CO2 error surfaces 

Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error surfaces 

Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error surfaces 

Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error surfaces 

Original

SwRI Report 03.12859.06 82 of 83 
12.0 REFERENCES 
 

1. 	 Feist, M.D., Sharp, C. A., Mason, R.L., and Buckingham, J.P., “Determination of PEMS 

Measurement Allowances for Gaseous Emissions Regulated Under the Heavy-Duty 

Diesel Engine In-Use Testing Program,” SwRI Project 03-12024, U.S. Environmental 

Protection Agency, California Air Resources Board and Engine Manufacturers 

Association, April 20. 

SwRI Report 03.12859.06	 83 of 83