# Additional Analyses of the Monte Carlo M o d e l D e v e l o p e d f o r t h e D e t e r m i n a t i o n of PEMS Measurement Allowances for Gaseous Emissions Regulated Under the H e a v y - D u t y D i e s e l E n g i n e I n - U s e T e s t i n g P r o g r a m

# Additional Analyses of the Monte Carlo M o d e l D e v e l o p e d f o r t h e D e t e r m i n a t i o n of PEMS Measurement Allowances for Gaseous Emissions Regulated Under the H e a v y - D u t y D i e s e l E n g i n e I n - U s e T e s t i n g P r o g r a m

Assessment and Standards Division Office of T ransportation and Air Quality U.S. Environmental Protection Agency
Prepared for EP A by Southwest Research Institute EP A Contract No. EP-C-05-018

W ork Assignment No. 2-6

NO TICE T

his tec hnical report does not necessarily represent final EP A decisions or positions. It is intended to present tec hnical analy sis of issues using dat a generated in the associated test program. T he purpose in the release of suc h reports is to f acilit ate the e x

c

hange of tec hnical inf ormation and to inf orm the public of tec hnical de v elopments whic h ma y f orm the basis f or a final EP A decision, position, or regulator y action.

EP A420-R-07-010 August 2007EXE CUTI V E SUMM ARY This repo rt docum ents the program conducted b y Southwest ResearchIn stitu te ® (Sw R I), on behalf of the U.S. Environm ental Protection Agency (EPA), the objectiv e of which was to perf orm addition al ana l yses on the Portable E m issions Measurem ent System s (PEMS) Monte Carlo s i mu lation m odels in o r der to de term ine validation of the thr ee em ission s (BSNO X , BSNMHC and BSCO) using three calculation m e thods. Several s t eady-state error surfaces were m odifi ed based on recomm endations from the Heavy-Duty In-Use Testing (HDIUT) Steering Comm ittee. These inclu d ed m odifications to the steady-state NO X , exhaust flow rate, CO and CO 2 error surfaces. Several reference NTE event s were run which produced validati on results from all three em i ssions and all three calculation m e thods. Measurem ent allowances were co mputed fo r two of the simulation strategies based on using 50 ref e rence NTE events

. The Mod 1 strategy involved changing the steady-state CO, CO 2 and exhaust flow rate error su rfaces to eliminate bias while changing th e steady -state NO X to a level independent error surface in cluding all th e tes tdata. The Mod 2 strategy was the sam e as Mod 1 except the stead y-state NO X error su rface was also change d to a level independent error surface but excluded s e veral qu estionable low NO X values from one of the test engines. The m easure m ent allowance values by calculation m e thod determ ined at the conclusion of these analyses are summ arized in Table 1

. This re port de ta ils the p r oces s used to d eterm ine th e m easure m ent values reported in Table 1

. TAB LE 1. MEASURE M ENT AL LO WANCES FOR M O D 1 AND MOD 2 Pollutan t Calculation Method Mod 1 Measurem ent Allowance, g/hp-hr Mod 2 Measurem ent Allowance, g/hp-hr NO X 1 0.232 0.211 2 0.178 0.151 3 0.192 0.171 NMHC 1 0.015 0.014 2 0.014 0.014 3 0.014 0.014 CO 1 0.268 0.266 2 0.258 0.250 3 0.270 0.262

Sw RI Repo r t 03 .1 285 9.06 iLIST OF ACRONYM S Brake-SpecificBS California A i r Resources Board CARB Center for E nvironm ental Research & Technology CE-CERT Code of Federal Regulations CFR Electronic F l ow Meter EFM Empirical D i stribution F unction EDF Engine Control Module ECM Engine Man ufacturer’s Association EMA Environm ental Protection Agency EPA Heavy Duty In-Use Testing HDIUT Heavy Heavy Duty HHD Light Heavy Duty LHD Median Absolute Deviation MAD Medium He avy Duty MHD Mem orandum of Agreem ent MOA Mobile Em issions Laboratory MEL Not To Exceed NTE Portable Emissions Measurem ent System PEMS Root Mean Square RMS SEMTECH- DS SN G05-SDS04 PEMS 1 SEMTECH- DS SN G05-SDS02 PEMS 2 SEMTECH- DS SN G05-SDS03 PEMS 3 SEMTECH- DS SN G05-SDS01 PEMS 4 SEMTECH- DS SN D06-SDS01 PEMS 5 SEMTECH- DS SN D06-SDS06 PEMS 6 SEMTECH- DS SN F06-SDS02 PEMS 7 Southwest Research Institute SwRI Standard Deviation SD

Sw RI Repo r t 03 .1 285 9.06 iiTABLE OF CONTENTS Page EXECUTIVE SUMMARY

............................................................................................................. i LIST OF ACRONYMS

............................................................................................................... ... ii LIST OF FI GURES

................................................................................................................ ....... iv LIST OF TABLES

................................................................................................................. ........ vi 1.0 INTRODUCTION

.............................................................................................................. 1 2.0 ERROR SURFACE MODIFICATIONS

............................................................................ 2 2.1 Steady-State Exhaust Flow Rate Error Surface

.................................................................. 2 2.2 Steady-State CO 2 Error Surface

.......................................................................................... 3 2.3 Steady-State CO Error Surface

........................................................................................... 4 2.4 Steady-State NO x Error Surface

.......................................................................................... 5 3.0 REFERENCE NTE EVENTS............................................................................................. 9 4.0 CE-CERT MEASUREMENTS

........................................................................................ 13 5.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS USING MOD 1, MOD 2 AND M O D 3

............................................................................ 14 6.0 MONTE CARLO VALIDATION RESULTS FROM 23 REFERENCE NTE EVENTS USING MOD B

................................................................................................................ 25 7.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS USING MOD 1

................................................................................................................. 35 8.0 MONTE CARLO VALIDATION RESULTS FROM 50 REFERENCE NTE EVENTS USING MOD 2

................................................................................................................. 45 9.0 MEASUREMENT ALLOWANCE CALCULATIONS

.................................................. 56 10.0 VALIDATION SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS.... 77 11.0 FULL MODEL SENSITIVITY RESULTS FROM 13 REFERENCE NTE EVENTS

... 81 12.0 REFERENCES

................................................................................................................ . 83 0

Sw RI Repo r t 03 .1 285 9.06 iiiLIST OF F I GURES Figure Page 1 Revised Error Surface for Stea dy-S t ate Exhaust Flow Rate ....................................3 2 Revised Error Surface for Steady-S t ate CO 2 ...........................................................4 3 Revised Error Surface for Steady-S t ate CO.............................................................5 4 Revised Error Surface Mod 1 for Steady-S t ate NO x ...............................................6 5 Revised Error Surface Mod 2 for Steady-S t ate NO x ...............................................7 6 Revised Error Surface Mod 3 for Steady-S t ate NO x ...............................................8 7 Distribution of Ideal NOx (g/kW

- h r) Method 1 for 23 Selected Ref NTE Events and 195 Ref NTE Events ...........................................................................11 8 Distribution of Ideal NOx (g/kW

- h r) Method 1 for 50 Selected Ref NTE Events and 195 Ref NTE Events ...........................................................................12 9 Validation E D F Plots Using 23 Reference NTE Events for NO x Method 1 Mods 1, 2 and 3 ......................................................................................................16 10 Validation E D F Plots Using 23 Reference NTE Events for NO x Method 2 for Mods 1, 2 and 3 ......................................................................................................17 11 Validation E D F Plots Using 23 Reference NTE Events for NO x Method 3 for Mods 1, 2 and 3 ......................................................................................................18 12 Validation E D F Plots Using 23 Reference NTE Events for NMHC Method 1 for Mods 1, 2 and 3 ................................................................................................19 13 Validation E D F Plots Using 23 Reference NTE Events for NMHC Method 2 for Mods 1, 2 and 3 ................................................................................................20 14 Validation E D F Plots Using 23 Reference NTE Events for NMHC Method 3 for Mods 1, 2 and 3 ................................................................................................21 15 Validation E D F Plots Using 23 Reference NTE Events for CO Method 1 for Mods 1, 2 and 3 ......................................................................................................22 16 Validation E D F Plots Using 23 Reference NTE Events for CO Method 2 Mods 1, 2 and 3 ................................................................................................................23 17 Validation E D F Plots Using 23 Reference NTE Events for CO Method 3 for Mods 1, 2 and 3 ......................................................................................................24 1 8 Validation E D F Plots Using 23 Reference NTE Events for NO x Method 1 Mod B .....26 1 9 Validation NTE Events Using 23 Reference NTE Events for NO x Method 2 Mod B ..27 2 0 Validation E D F Plots Using 23 Reference NTE Events for NO x Method 3 Mod B .....28 21 V a li d a t i o n E

D F P

l ot sU s i ng 2 3 R e f

e

r enc e N T

E

Ev en t sf o

r N M H C M e t ho d 1 M o d B ..29 22 V a li d a t i o n E

D F P

l ot sU s i ng 2 3 R e f

e

r enc e N T

E

Ev en t sf o

r N M H C M e t ho d 2

M o d B ..30 23 V a li d a t i o n E

D F P

l ot sU s i ng 2 3 R e f

e

r enc e N T

E

Ev en t sf o

r N M H C M e t ho d 3 M o d B ..31 2 4 Validation E D F Plots Using 23 Referen ce NTE Events for CO Method 1 Mod B ......32 2 5 Validation E D F Plots Using 23 Referen ce NTE Events for CO Method 2 Mod B ......33 2 6 Validation E D F Plots Using 23 Referen ce NTE Events for CO Method 3 Mod B ......34 2 7 Validation E D F Plots Using 50 Reference NTE Events for NO x Method 1 Mod 1 .....36 2 8 Validation E D F Plots Using 50 Reference NTE Events for NO x Method 2 Mod 1 .....37 2 9 Validation E D F Plots Using 50 Reference NTE Events for NO x Method 3 Mod 1 .....38 30 V a li d a t i o n E

D F P

l ot sU s i ng 5 0 R e f

e

r enc e N T

E

Ev en t sf o

r N M H C M e t ho d 1 M o d 1 ...39 31 V a li d a t i o n E

D F P

l ot sU s i ng 5 0 R e f

e

r enc e N T

E

Ev en t sf o

r N M H C M e t ho d 2

M o d 1 ...40 32 V a li d a t i o n E

D F P

l ot sU s i ng 5 0 R e f

e

r enc e N T

E

Ev en t sf o

r N M H C M e t ho d 3 M o d 1 ...41 3 3 Validation E D F Plots Using 50 Referen ce NTE Events for CO Method 1 Mod 1 .......42 3 4 Validation E D F Plots Using 50 Referen ce NTE Events for CO Method 2 Mod 1 .......43

Sw RI Repo r t 03 .1 285 9.06 iv3 5 Validation E D F Plots Using 50 Referen ce NTE Events for CO Method 3 Mod 1 .......44 3 6 Validation E D F Plots Using 50 Reference NTE Events for NO x Method 1 Mod 2 .....46 3 7 Validation E D F Plots Using 50 Reference NTE Events for NO x Method 2 Mod 2 .....47 38 Validation E D F Plots for 50 Reference NTE Events for NO x Method 3 Mod 2 ...48 3 9 Validation E D F Plots for 50 Reference NTE Events for NMHC Method 1 Mod 2 .....49 4 0 Validation E D F Plots for 50 Reference NTE Events for NMHC Method 2 Mod 2 .....50 41 V a li d a t i o n E

D F P

l ot sU s i ng 5 0 R e f

e

r enc e N T

E

Ev en t sf o

r N M H C M e t ho d 3 M o d 2 ...51 4 2 Validation E D F Plots Using 50 Referen ce NTE Events for CO Method 1 Mod 2 .......52 43 Validation E D F Plots Using 50 reference NTE Events for CO Method 2 Mod 2 .53 4 4 Validation E D F Plots Using 50 Referen ce NTE Events for CO Method 3 Mod 2 .......54 4 5 Regression Plot of 95 th Percentile Delta BSNOx Versus Ideal BSNO x f

or Method 1 M o d 1......................................................................................................................57 4 6 Regression Plot of 95 th Percentile Delta BSNOx Versus Ideal BSNO x f

or Method 2 M o d 1......................................................................................................................58 4 7 Regression Plot of 95 th percentile Delta BSNOx Versus Ideal BSNO x f

or Method 3 M o d 1......................................................................................................................59 4 8 Regression Plot of 95th percentile Delta BSNMHC Versus Ideal BSNMHC for Method 1 M od 1 ......................................................................................................60 4 9 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for Method 2 M od 1 ......................................................................................................61 5 0 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for Method 3 M od 1 ......................................................................................................62 5 1 Regression Pl ot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 1 M o d 1 .....................................................................................................................63 5 2 Regression Pl ot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 2 M o d 1 .....................................................................................................................64 5 3 Regression Pl ot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 3 M o d 1 .....................................................................................................................65 5 4 Regression Plot of 95 th Percentile Delta BSNO X

Versus Ideal BSNOX f

o

r Method 1 M o d 2......................................................................................................................66 5 5 Regression Plot of 95 th Percentile Delta BSNOx Versus Ideal BSNO x f

or Method 2 M o d 2......................................................................................................................67 5 6 Regression Plot of 95 th Percentile Delta BSNOx Versus Ideal BSNO x f

or Method 3 M o d 2......................................................................................................................68 5 7 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for Method 1 M od 2 ......................................................................................................69 5 8 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for Method 2 M od 2 ......................................................................................................70 5 9 Regression Plot of 95th Percentile Delta BSNMHC Versus Ideal BSNMHC for Method 3 M od 2 ......................................................................................................71 6 0 Regression Pl ot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 1 M o d 2 .....................................................................................................................72 6 1 Regression Pl ot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 2 M o d 2 .....................................................................................................................73 6 2 Regression Pl ot of 95th Percentile Delta BSCO Versus Ideal BSCO for Method 3 M o d 2 .....................................................................................................................74 0

Sw RI Repo r t 03 .1 285 9.06 vLIST OF TABLES Table Page 1 Measurem ent Allowances For MOD 1 and Mod 2

.................................................. i 2 Revised Error Surfaces Used in Mo nte Carlo S imulations

...................................... 8 3 Reference NTE Events Used in Monte Carlo S imulations

.................................... 10 4 Descriptive Statis tics for Ideal BSNO x (g/kW

- hr) for Various NTE Subs ets
....... 12 5 Convergence criteria by Em ission

......................................................................... 14 6 BSNO x Validation Results Based on 23 Reference NTE Events

.......................... 15 7 Summ ary of Validation Results

............................................................................. 55 8 Measurem ent Error at T h reshold for BSNO x Using Regression and Median Methods for Method 1 Mod 1

................................................................................ 57 9 Measurem ent Error at T h reshold for BSNO x Using Regression and Median Methods for Method 2 Mod 1

................................................................................ 58 10 Measurem ent Error at T h reshold for BSNO x Using Regression and Median Methods for Method 3 Mod 1

................................................................................ 59 11 Measurem ent Error at T h reshold for BSNMHC Using Regression and Median Methods for Method 1 Mod 1

................................................................................ 60 12 Measurem ent Error at T h reshold for BSNMHC Using Regression and Median Methods for Method 2 Mod 1

................................................................................ 61 13 Measurem ent Error at T h reshold for BSNMHC Using Regression and Median Methods for Method 3 Mod 1

................................................................................ 62 14 Measurem ent Error at T h reshold for BSCO Using Regression and Median Methods for Method 1 Mod 1

................................................................................ 63 15 Measurem ent Error at T h reshold for BSCO Using Regression and Median Methods for Method 2 Mod 1

................................................................................ 64 16 Measurem ent Error at T h reshold for BSCO Using Regression and Median Methods for Method 3 Mod 1

................................................................................ 65 17 Measurem ent Error at T h reshold for BSNO x Using regression and Median m e thods for Method 1 Mod 2

................................................................................ 66 18 Measurem ent Error at T h reshold for BSNO x Using Regression and Median m e thods for Method 2 Mod 2

................................................................................ 67 19 Measurem ent Error at T h reshold for BSNO x Using Regression and Median Methods for Method 3 Mod 2

................................................................................ 68 20 Measurem ent Error at T h reshold for BSNMHC Using Regression and Median Methods for Method 1 Mod 2

................................................................................ 69 21 Measurem ent Error at T h reshold for BSNMHC Using Regression and Median Methods for Method 2 Mod 2

................................................................................ 70 22 Measurem ent Error at T h reshold for BSNMHC Using Regression and Median Methods for Method 3 Mod 2

................................................................................ 71 23 Measurem ent Error at T h reshold for BSCO Using Regression and Median Methods for Method 1 Mod 2

................................................................................ 72 24 Measurem ent Error at T h reshold for BSCO Using Regression and Median Methods for Method 2 Mod 2

................................................................................ 73

Sw RI Repo r t 03 .1 285 9.06 vi25 Measurem ent Error at T h reshold for BSCO Using Regression and Median Methods for Method 3 Mod 2 ................................................................................74 26 Summ ary of Measurem ent Errors at Respective Threshold (%) ...........................75 27 Summ ary of Measurem ent Allowance, g/hp-hr .....................................................76 28 Sensitiv ity Results Co mparing 13 Ref e rence NTE Events Across 5 M onte Carlo validation Simulations for BSNO x ...............................................................78 29 Sensitiv ity Results Co mparing 13 Ref e rence NTE Events Across 5 M onte Carlo validation Simulations for BSNMHC ..........................................................79 30 Sensitiv ity Results Co mparing 13 Ref e rence NTE Events Across 5 M onte Carlo validation Simulations for BSCO ................................................................80 31 Sensitiv ity Results Co mparing 13 Ref e rence NTE Events Across 5 M onte Carlo f u ll model Simu lations f o r BSNO x ..............................................................82

Sw RI Repo r t 03 .1 285 9.06 vii1.0 INTRODUCTION Southwest Research In stitu te comp leted a Po rtab le Em issions Meas urem ent System (PEMS) program on behalf of the U.S. Environm ental Pro t ection Agency (EPA) in early Apri l 2007. The purpose of this project was to dete rm ine the brake-specific (BS) m easurem ent allowances for the gaseous pollutants regulatedunder the Heavy-Duty In-Use Testing (HDIUT) program [Ref 1]. The s t udy was perform e d under cooperation between the EPA, the California Air Resources Board (CARB), and the Engine Manufacturer’s Asso c i ation (EMA). All efforts during this program were conducted under the direction of a joint body, the HDIUT Measurem ent Allowance Steering Comm ittee, referred to in this repo rt sim p ly as the Stee rin g Comm ittee. The program consisted of m odeling various PE MS m easurem ent errors using a sta t istical Monte Carlo m odeling simu lation. The si mu lation results were used to generate the brakespecific m easurem ent allowances based on thre e calculation m e thods fo r BS em issions of NO x , NMHC and CO. To confirm the results of th e simulation, a SEMTECH- DS PEMS was operated in-use with the CE-CERT Mobile E m ission Laborat ory (ME L ) over seve ral ro utes in California. The differences between the PEMS and MEL ga seous em ission m e asurem ents were used to validate the SwRI Monte Carlo m odeling simulation results. Calculation Methods 2 (BSFC based) and 3 (ECM Fuel Specific) did not validate for BSNO x base d on the CE-CERT MEL e m issions and th e Monte Carlo PE MS simu lations. Given this result, EPA contra cted with S w RI in April 2007 to co nduct additional anal yses to identify possible cau ses as to wh y the BSNO x did not validate. The results of these analyses are includ ed in this repo rt.

Sw RI Repo r t 03 .1 285 9.06 1 of 832.0 ERROR S URFACE MODIFICATIONS Since calculation Methods 2 and 3 did not validate for BSNO x after the Monte C arlo model simu l a tion runs d u ring the original PEMS study, the associated Steering Comm ittee chose to m odify key error su rfaces in or d e r to d eterm ine if su ch alterations would effect th e validation of Methods 2 and 3. Over the cours e of seve ral m eetings, the Steering C o mm ittee brainsto rm ed to generate alternate error su rface processing m e thods. The Comm i ttee decided to m odify Steady-State CO, CO 2 , Exhaust Flow Rate, and NO x , as these surfaces had significant influen c e on the Mode l resu lts b as ed on the se nsitiv ity ana l yses. In reproc es sing the error surfaces, the Steerin g Comm ittee agreed th at it wouldbe necessary to rem ove any bi as es that were record ed in the SwRI laboratory. It was assum e d that these bia s errors were d u e to the lim ited num b er of PEMS units and com p arative observations, and that if more PEMS and Sensors Inc. e xhau st flow m e ters were tes t ed the b i as es may have been elim inated. Therefore, to remove the b i as es the orig inal s t eady-s t ate error su rfaces were transform ed to be a symm etric error surface withthe 50 th per centile error s set to ze ro and the 5 th and 95 th percentile errors m odified to be a m i rror im age of one a nother. Two analysis m e thods were used to generate the rev i se d error surfaces

. If th e reco rded error d ata showed an em issions level dependency, an envelope was ge nerated to encompass the extrem e 5 th or 95 th percentile error d ata. The envelop e segm ents with the larg es t abso lute error were then m i rrored to genera te a symm etric erro r su rface. A m o re rigo rous an al ysis was u sed to rep r o cess level independen t error surface s. 2.1 Steady-State Exhaust FlowRate Er ror Surface Figure 1 shows the reprocessed, level-dependent steady

- state exhaus t flow rate error surface as well as the exhaust flo w error surface da ta th at was used in the origin al MC m o del runs. The largest e rro rs record ed during laboratory testing w e re positive deltas m easured during Engine 3 testing withth e 3-inch EFMs. The 3-inch EFM errors definedthe 95 th p ercentile e rro r contours for the original error su rface as well as the rep r ocesse d erro r s urface. The difference between the two 95 th percentile con t ours was due to a corr ection of the 3-inch EFM calibrations when the exhaust flow rate error surface was reproces sed. SwRI and Sensors Inc. had reca libr atedthe 3

- in chf l ow m eters us ing da ta gen era te d at SwRI. The SwRI ca lib ratio n incre ased th e slope mu ltiplier by ap proxim ately 4 % f o r ea ch 3-in ch f l ow m eter. Later in the program , Sensors Inc. discovered the 3-inch EFMs were likely not ope r ating corr ec tly on the SwRI flow stand due to EFM pressu res below ambi ent levels. Therefore, each 3-inch flow m eter calib ration was corrected to the orig inal coefficients generated by Sensors. The corrected 3-inch flow m e ter data generated the 95 th percentile contour shown in Figure 1

. Because the 95 th percen tile errors were larger than the 5 th percentile errors, the 95 th percentile data was m i rrored to generate the 5 th percen tile contour in the rep r oces sed erro r surface.

Sw RI Repo r t 03 .1 285 9.06 2 of 83-6 -4 -2 0 2 4 6 8 10

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8

Lab R e fe rence M e an Exhaust Flow R a te [% of M a x]PEMS Exhaust FlowRate Delta[%ofMax] 95 t h pe rc e n t ile 50t h perc e n t ile (m e d ian) 5 t h pe rc e n t ile P r evious 9 5 t h P r ev iou s 50 t h Prev ious 5t h

FIGURE 1. REVIS E D ERROR S URFACE F O R STEADY-STATE EXHAUS T FLOW RAT E 2.2 Steady-State CO 2 Erro r Surface Figure 2 shows the reprocessed, level-dependent CO 2 error surface. The 95 th per cent i l e contour represents the enve lope th at was createdto encom p ass the o r igin al 95 th percentile erro r data. Line segm ents were used to connect the m ost extrem e points from the original 95 th percen tile d ata. The 95 th percentile contour was m i rrored to produce the 5 th percen tile erro r data, with the 50 thpercen tile d eltas set to zero. The repro cessed error surface was m eant to encomp ass all possible PEMS CO 2 m easurem ent erro rs, not just tho s e measured at S w RI. 0

Sw RI Repo r t 03 .1 285 9.06 3 of 83-0. 8 -0. 6 -0. 4 -0. 2 0. 0 0. 2 0. 4 0. 6 0. 8

4

. 0 4. 5 5. 0 5

. 5 6. 0 6. 5 7

. 0 7. 5 8. 0 8

. 5 9. 0

Lab R e fe re n c e M e an C O 2 C oncentra t i on [% ]PEMSCO2 Concentration Delta [%] 95 t h pe r c e n t i l e 50 t h per c e nt ile (m edian ) 5t h p e rc ent ile Prev ious 95t h P r evio us 5 0 t h P r ev ious 5t h

FIGURE 2. REVIS E D ERROR S URFACE F O R STEADY-STAT E CO 2 2.3 Steady-State CO Error Surface Figure 3 shows the revis e d steady-state error surf ace for CO. Because the CO error data showed no level dependence, a statistical calcul ation process, develope d by Bill Martin from Cumm ins I nc., was used to generate the repro cessed CO error surfa ce. The to tal s t andard deviation of the pooled CO error data was calcu lated by combining the variance of the PEMS m ean delta standard deviation a bout zero and the PEMS pooled estim ate of re peatability standard deviation. T he calculation of the tota l standard deviation is shown below.

SD indiv, pems = standard d e viation of each PEMS calcu lated over 20 repeats of each steady-state point

n = 10 steady-state points per engi ne * 3 engines * 3 PEMS = 90 2

, 2 zero mean ity repeatabil total SD SD SD + =

n SD SD n pems indiv ity repeatabil 

## ∑

= 1 2

,

Sw RI Repo r t 03 .1 285 9.06 4 of 83PEMS indiv, mean = m ean delta of each P E MS calculated over 20 repeats of each steady-s t ate po int Shown below, the multip lie r f o r

a 90% c onf idence interva l ass u m i ng a norm al distribution for the errors was applied to th e total standard deviation (48.84 ppm = 0.004882%) to generate the constant 5 th and 95 th percen tile error valu es.

-0. 010 -0. 008 -0. 006 -0. 004 -0. 002 0. 000 0. 002 0. 004 0. 006 0. 008 0. 010 0. 012

0. 0000 0. 0005 0. 0010 0. 0015 0. 0020 0. 002 5

L a b R eference M ea n C O C o ncentrati on [% ]PEMSCOConcentrationDelta[%] 95 t h pe r c e n t i l e 50 t h pe rc e n t ile (m edian ) 5t h p e rc ent ile Prev ious 95t h P r evious 5 0 t h P r ev iou s 5t h

For reprocessing the or iginal s t ead generate three NO x concentration error surfaces. For each of the three m odi fied NO x error surfaces, the original set of Engine 2 steady-state data was added to the origin al pooled data set. The Engine 2 steady-state data was collected in a repeat run in the orig inal study due to PEMS failures th at caused a large tim e gap in the original data. The tim e gap was associated with a NO x concentra tion sh if t. Since the NO x shift would have caused pr oblem s with th e steady

- state 

# ( )
1 0 1 2

,

,

− −

= 

## ∑

n PEMS SD n

mean indiv

zero mean

% 008

. 0 * 645

. 1 % 95 % 008

. 0 * 645

. 1 % 5 + = = − = − =
total th total th SD SD

FIGURE 3. REVIS E D ERROR S URFACE F O R STEADY-STAT E CO

2.4 Steady-State NO x Erro r Surface y-state NO x , three different m e thods were used to

Sw RI Repo r t 03 .1 285 9.06 5 of 83varian ce correction of the trans i en t data th e repeat run d a ta was used. However, the original Engine 2 delta da ta was later de te r m ined to be valid and thus was included in the modified data set to increase the PEMS delta observations.
steady-state CO error su rface. Usin g the comp lete NO x delta data se t, the 5 th and 95 th percen tile error values were calculated wi th the total standard deviation (15.08 ppm )

. The results of the fi rstNO x error surface m odification are shown in Figure 4 with the 5 th and95 th perc entiles set to ± 24.80 ppm.

-1 00 -8 0 -6 0 -4 0 -2 0

0 20 40

10 0 20 0 30 0 40 0 50 0 6 00 7 00

La b R e f e r e n c e M e a n N O xC o n c e n t r a t io n [ p pm]PEMS NOxConcentration Delta[ppm] 95 t h pe rc en t i l e 50 t h pe r c en t i l e ( m ed i an ) 5th p e r c e n t i le P r evi o u s 95 t h P r evi o u s 50 t h P r evi o u s 5t h

except s i x o u tlying low delta v alues record ed d u ring Engine 3 testing were rem oved from the pooled data set. During Engi ne 3 steady-state testing, PEMS 1 and 6 showed extrem ely low de l t a s a thi gh NO x concentration levels. PEMS 4 al so sh owed low biases at hig h NO x levels; however, th e biases were not as larg e as tho se measured for the other PEMS. Therefore, PEMS 1 and 6 delta m easurements at the three highest NO x levels m easured during engine testing were rem oved from the data set. This resulted in a total standard deviation equal to 9.39 ppm and the 5 th and 95 thpercen tile delta s equa l to ± 15.45 ppm

. The results of th is analysis are shown in Figure 5

. The first NO x error surface m odifi cation (Mod1) followed a procedure sim ilar to the

FIGURE 4. REVIS E D ERROR S URF ACE MOD 1 FO R STEADY-STATE NO X

The second NO x error s u rface m odi fication (Mo d 2) was sim ilar to the first m odification

Sw RI Repo r t 03 .1 285 9.06 6 of 83-1 00 -8 0 -6 0 -4 0 -2 0

0 20 40

10 0 20 0 30 0 40 0 50 0 6 00 7 00

La b R e f e r e n c e M e a n N O xC o n c e n t r a t io n [ p pm]PEMS NOxConcentration Delta[ppm] 95 t h pe rc en t i l e 50 t h pe r c en t i l e ( m ed i an ) 5th p e r c e n t i le P r evi o u s 95 t h P r ef i ou s 50 t h P r evi o u s 5t h

FIGURE 5. REVIS E D ERROR S URF ACE MOD 2 FO R STEADY-STATE NO X Shown in Figure 6 , the third NO x error surface modification (M od 3) combined the leveldependent and level-independent analysis m e thods

. Below 311 ppm , the delta data w as assum ed to be level independent. Therefore, a statistical m e thod si m i lar to that used for CO was used for the NO x delta data below 311 ppm. The total standard deviation for the level-independent data was 9.41 ppm and the resulting 5 th and 95 th percentiles were ± 15.48 ppm. Above 311 ppm , the origin al 5 thpercentile profile was used and m i rrored to the 95 th percen tile to generate a symmetric error su rface.

Sw RI Repo r t 03 .1 285 9.06 7 of 83-1 00 -8 0 -6 0 -4 0 -2 0

0 20 40 60 80 10 0

10 0 20 0 30 0 40 0 50 0 6 00 7 00

La b R e f e r e n c e M e a n N O xC o n c e n t r a t io n [ p pm]PEMS NOxConcentration Delta[ppm] 95 t h pe rc en t i l e 50 t h pe r c en t i l e ( m ed i an ) 5th p e r c e n t i le P r evi o u s 95 t h P r evi o u s 50 t h P r evi o u s 5t h

FIGURE 6. REVIS E D ERROR S URF ACE MOD 3 FO R STEADY-STATE NO X

In addition to running the MC simul a tions wi th the three different changes to the steadystate NO X error su rface, it was us eful to run a s i mulation with the origin al steady-state NO X error surface andonly include the revis e dsteady

- st ate exhaust flo w rate and steady-state CO 2 error surfaces. T h is set of simulations is called the Mod B runs. Table 2 provides a summary of the four MC si mulation runs m a de in this study with the revised error surfaces that were included in each run. All the oth er error su rfaces that are not listed in Table 2 are the sam e as those run in the original PEMS study.

TAB LE 2. REVIS E D ERROR S URFACES US ED I N MONTE CARLO SIM U LATIONS Error Surfac e Mod B Run Mod 1 Run Mod 2 Run Mod 3 Run Steady-State Exhaust Flow Rate X X X X Steady-State CO 2 X X X X Steady-State CO X X X Steady-State NO x Mod 1 X Steady-State NO x Mod 2 X Steady-State NO x Mod 3 X

Sw RI Repo r t 03 .1 285 9.06 8 of 833.0 REFERENCE NTE EVENTS The Monte Carlo simu lation results from the original study included a reference data set consisting of 195 NTE events gathered from a nu m ber of sources. These included five engine m anuf acture r s, SwRI transien t lab tests and pre-pilot C E -CERT data. The investig ation s perform e d i n this m odified program included on ly a subset of the original 195 reference NTE events. Simulations from three different subsets of the original NTE eve nts were run to study the validation of the em issio n s by the th ree calculatio n m ethods using the revised erro r surfaces. Table 3 lis ts the NTE events alon g with th eir idea l BSNO x values s elec t ed f o r each simulation. The various NTE subs ets are described as follows:

• 13 Reference Events – These are the sam e 13 NTE events chosen during the original study to inv estig ate the error su rface sensitivities due to bias and variance. These events were selected to bound the BSNO x threshold of 2.6820 g/kW -hr. Thes e 13 NTE events were used to exam ine the erro r surf a ce sensitiv ities f o r both the valida tio n and f u ll model simu lation runs. • 23 Reference Events – Ten addi tio nal NTE ev ents wer e a dded to the origin al 13 NTE events d e scribed abov e in order to in creas e th e s a mp le size f o r generatin g the v a lidation EDF plots using the drift corrected C E -CERT data. • 50 Reference Events – After the model valida tion was confirm e d for the three em issions across the three calculation m e thods, Mods 1 a nd 2 were chosen for continued analysis. Twenty-sev en additional reference NTE even ts were simu lated with th e Monte Carlo model in order to estimate the m easurem ent allo wances for each em ission. The selection of the add itional 27 reference NTE events wa s based on m a intaining a similar distribution of ideal BSNO x values for Method 1 established from the original study containing 195 reference N TE events. Figure 7 shows compa r ison histogram s of the Method 1 BSNO x va lues for the 23 reference N TE events (lower histogram ) and the 195 reference NTE events (upper histogram ) while Figure 8 compares the Meth od 1 BSNO x histogram s for the 50 reference NTE events (lower histogram ) and t h e 195 reference NTE e v ents (upper histogram ). Descriptive statistics of BSNO x for these NTE subsets are detailed in Table 4

.

Sw RI Repo r t 03 .1 285 9.06 9 of 83TABLE 3. REFERENCE N TE EVENTS US ED IN MONTE CARLO SIMULATIONS NTE # Ideal NOx (g/kW

- hr) Method 1 13 Reference Events 23 Reference Events 50 Reference Events 1 4.0713 X X X 3 3.0668 X X 4 3.5832 X 7 5.2516 X X 11 1.8583 X 16 2.3511 X 20 3.7245 X X 22 4.7916 X 23 3.1483 X 25 5.4061 X X X 29 5.5261 X X 37 5.4511 X 38 0.0250 X X X 40 4.1675 X 43 1.1473 X 44 1.0730 X X X 46 2.6958 X X X 51 2.8299 X X X 57 4.3382 X X 63 2.6670 X X X 65 2.7437 X 66 3.9378 X 67 5.8600 X 69 3.0257 X X X 71 6.6867 X 82 2.4569 X X X 86 1.7132 X 87 1.5207 X X X 89 2.2566 X X 92 1.6041 X 96 1.6224 X X 99 1.8147 X 103 1.9186 X 115 1.3854 X 125 2.3214 X 127 3.3005 X 136 2.6782 X X 139 2.4018 X X 146 2.3053 X 148 1.9985 X X X 157 3.4666 X X X

Sw RI Repo r t 03 .1 285 9.06 10 of 83160 2.0773 X 162 2.1405 X 163 2.5908 X X X 168 1.5859 X 176 1.9671 X X 177 1.7507 X 179 2.2023 X 191 6.0815 X 193 5.7521 X



## I d eal NO x M e t h od 1 195 R e f N T E

I d eal NO x M e t h od 1 23 R e f N T EFrequency



### 0 2 4 6 8 20 10

0 10 20 30 40

FIGURE 7. DISTRI B U TION OF ID EAL NOX (G/KW-HR) METHOD 1 FOR 23 SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS

Sw RI Repo r t 03 .1 285 9.06 11 of 83

# Ideal NOx Method 1 195 Ref NTE Events

Ideal NOx Method 1 50 Ref NTE EventsFrequency



### 0 2 4 6 8 20 10

0 10 20 30 40

FIGURE 8. DISTRI B U TION OF ID EAL NOX (G/KW-HR) METHOD 1 FOR 50 SELECTED REF NTE EVENTS AND 195 REF NTE EVENTS

TAB LE 4. DESCRIPT IVE ST ATI STICS FO R IDE A L B SNO X (G/KW-HR) FOR VARIOUS NTE S U BS ETS Descriptive Statis tic 23 Reference NTE Events 50 Reference NTE Events 195 Reference NTE Events Minimu m 0.0249 0.0249 0.0249 Maximu m 5.5261 6.6867 7.1927 Mean 2.8983 3.0068 3.0071 Median 2.6782 2.6289 2.6033 Standard Deviation 1.3671 1.5175 1.4807

Sw RI Repo r t 03 .1 285 9.06 12 of 834.0 CE-CERT MEASUREMENTS The CE-CERT on-road m easure m ents collect ed during the original PEMS study included delta BS emissions for all three em issions and three calculation m e thods. As part of this study to investigate a ll po ssi ble reasons why BSNO x did not validate for Me thods 2 and 3, the 100 NTE events from the CE-CERT data were exam inedfor correctness by EPA. Since the CE-CERT data had not been drift corrected in the origin al PEMS study, EPA perform e d drift corrections on all three emissions for the 100 CE-CERT NTE events. As a result, several CE-CERT NTE events d i d n o t pass the drift check criteria and, theref ore, were not in cluded in th e simu lation perform ed for th is study

. In add itio n, tim e alig nm ent problem s were found in a few of the 10 0 CE-CERT NTE events and these were also excl uded from the simul a tion. After deleting the NTE events due to drift check or tim e alignm ent problem s, the on -road delta BS em issions were calcu latedfrom 81 NTE events for B S NO x and 87 NTE events for BSNMHC and BSCO. Thes e rem aining CE-CERT NTE events were all d r if t corrected. In contrast, the original PEMS program did not use drift correction in the CE-CERT on-road NTE events.

Sw RI Repo r t 03 .1 285 9.06 13 of 835.0 MONTE CARLO VALIDATION RESU LTS FROM 23 REFERENCE NTE EVE N TS USING MOD 1, MOD 2 AND MO D 3 The Monte Carlo simu lations perform e d usi ng the 23 reference NTE events listedin Table 3 included m odifi cation s for the erro r su rfaces lis ted in Table 2

. All the err o r surf aces f r om the original MC simulations w ere used exc ept those lis ted as rev i se d in Table 2

. Each of the 23 reference NTE events were run using th e erro r surfaces revised for Mod 1 for eithe r 10,000 or 30,000 trials. If the ideal BSNO X for an NTE e v ent was less than 2.68204 g/kW -hr then the simu lation was run for 10,000 trials. Otherwise, the NTE was simulated using 30,000 trials. Once the MC simulations w e re comp leted using the Mod 1 revised erro r su rfaces, the sam e 23 NTE events were simu lated a second tim e with the error surface m odifications for Mod 2. Lastly, the 23 reference NTE events were s imulated using the error surface m odifications for Mod 3. In Mod 1, Mod 2, and M od 3 si mu lation runs, all three em issions converged within 1% of the em issions threshold value by all three calculation m e thods. Table 5 lists the convergence criter ia f o r all th ree br ake-spe cif ic em issions. Thus, all 23 ref erence NTE events m et the convergence criteria.
TAB LE 5. CONVE RGENCE CRI TERIA BY EMISSION Em ission 1% of Threshold, g/kW -hr BSNOx 0.02682 BSNMHC 0.00282 BSCO 0.26015

From the MC simu latio ns, the valid ation 5 th and95 th percentile delta BS em issions were extra cted f r o m the outp u t f iles f o r e ach of the 2 3 reference NTE events. These d e lta em issions were then plotted as empirical distribution functions (EDF) to fo rm a valida tion inte rval f o r

th e on-road data. Also plotted was the EDF comp uted from the on-road CE-CERT NTE events. Figure 9 through Figure 11 represent the validation plots for the BSNO x for calculation m e thods 1, 2 and 3, respectively. Each of th e valid ation plots includes 5 th and 95 th percen tile E D Fs f o r the Mod 1, Mod 2, and Mod 3 runs. Figure 12 through Figure 17 depict the validation plots for the BSNMHC and BSCO simulation ru ns, respectively. Th e va lida tion criteria set by the Steering Comm ittee for the o r ig inal study was used in th e m odified program

. It included the following criteria:

• At leas t 90 % of the CE-CERT em issi ons delta s mu st be within the 5 th and 95 th percentiles of the MC validati on cumulative em issions deltas. • No m o re than 10% of the CE-CERT em issions deltas m ay f all less than the 5 th percen tile or greater than the 95 th p er centile. Th is m ay indicate tha tthe model is bias e d low or high. • Validation must be shown for all three calculation m e thods.

Sw RI Repo r t 03 .1 285 9.06 14 of 83A summ ary of the BSNO x validation conclusions based onthe 23 reference NTE events is provided in Table 6

. All BSNO X , BSNMHC and BSCO e m issions validated for all three calculation m e thods and all three Mod runs.
TABLE 6. BSNO X VALIDATION RESULTS BASE D ON 23 REFE RENCE NT E EVENTS
Simu lation Run Calculation Method 1 2 3

Mod 1 · All CE-C ERT data > 5th percentile · All CE-C ERT data < 95th percentile

· VALID · All CE-C ERT data > 5th percentile · 1% of CE-CERT data > 95th percentile

· VALID · All CE-C ERT data > 5th percentile · All CE-C ERT data < 95th percentile

· VALID

Mod 2 · All CE-C ERT data > 5th percentile · All CE-C ERT data < 95th percentile

· VALID · All CE-C ERT data > 5th percentile · 1% of CE-CERT data > 95th percentile

· VALID · All CE-C ERT data > 5th percentile · 1% of CE-CERT data > 95th percentile

· VALID

Mod 3 · All CE-C ERT data > 5th percentile · All CE-C ERT data < 95th percentile

· VALID · All CE-C ERT data > 5th percentile · All CE-C ERT data < 95th percentile

· VALID · All CE-C ERT data > 5th percentile · All CE-C ERT data < 95th percentile

· VALID

Sw RI Repo r t 03 .1 285 9.06 15 of 83FIGURE 9. VALI DAT ION EDF P L OTS USI N G 23 REF E RE NCE NTE EVE N TS FOR NO X METHOD 1 MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 16 of 83FIGURE 1 0. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NO X ME T HOD 2 FOR MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 17 of 83FIGURE 1 1. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NO X ME T HOD 3 FOR MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 18 of 83FIGURE 1 2. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 1 FOR MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 19 of 83FIGURE 1 3. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 2 FOR MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 20 of 83FIGURE 1 4. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 3 FOR MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 21 of 83FIGURE 1 5. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 1 FOR MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 22 of 83FIGURE 1 6. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 2 MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 23 of 83FIGURE 1 7. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 3 FOR MODS 1, 2 AND 3

Sw RI Repo r t 03 .1 285 9.06 24 of 836.0 MONTE CARLO VALIDATION RESU LTS FROM 23 REFERENCE NTE EVENTS USING MOD B Since all three m e thods validated using the 23 reference NTE events by all three modification runs for BSNO x , BSNMHC and BSCO, the question arose as to which of the revisederro r surfaces h ad the m o st influence in the validation. Three possible ex planations included(1) the s t eady

- state CO 2 bias had been elim inated, (2) the ch a nge in the s t eady-s t a t e NO X error values, and (3) the bias in the steady

- stat e exhaust flow rate had been elim inated. To study these possible explanations , none of the revised SSNO x error su rfaces or the SSCO error surface was used in the s i mu lations. These error surfaces were set to their orig inal form at in the original PE MS study. Therefore, only the stea dy-state exhaust flow ra te and the steady-state CO 2 error s u rfaces were revis ed an d the other re m a ining validation error surfaces were set to their original definitions for running the 23 reference NTE events for the Mod B MC simulation. This set of simu lations is ref err e d to as the ‘Mod B’ runs. Again, the simu lations were perform e d at 10,000 trials for NTE events with ideal BSNOx values less than 2.68204 g/kW -hr and at 30,00 trials otherwise. All 23 reference N TE events m e t the convergence criteria listed in Table 5

. From the Mod B MC simulations, the validation 5 th and 95 th per centile d e lta BS em issions were ex tracted from the output files for each of the 23 reference NTE events. The s e delta em issions were th en plotted as empirical di stribution functions (E DF ) to f o rm a va lida tion interval for the on-road data. Also plotted was the EDF computed from the on-ro ad CE-CERT NTE events. Figure 18 through Figure 20 represent the valida tion p l ots f o r the BSNO x for calculation m e thods 1, 2 and 3, respectively. Each of the validati on plots includes 5 th and 95 th percentile E D Fs for the Mod B runs and the on-road CE-CERT data. Figure 21 through Figure 26 depict the validation plots for the BSNMHC and BSCO simu lation runs, respectively. All BSNOx, BSNMHC and BSCO e m is sions validated fo r all three calculation m ethods for the Mod B runs. Conclusions m a de from the results of the Mod B analys is were th at the s t eady-state NO X error surface changes did not have as mu ch of an effect on t h e validatio n as the bias elim ination in the s t eady

- state CO 2 and steady-state exhau st flow rate erro r surfaces.

Sw RI Repo r t 03 .1 285 9.06 25 of 83FIGURE 1 8. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NO X METHOD 1 MOD B

Sw RI Repo r t 03 .1 285 9.06 26 of 83FIGURE 1 9. VALIDATION NT E EVENT S USING 23 REFERENCE NTE EVENTS FO R NO X METHOD 2 MOD B

Sw RI Repo r t 03 .1 285 9.06 27 of 83FIGURE 2 0. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NO X METHOD 3 MOD B

Sw RI Repo r t 03 .1 285 9.06 28 of 83FIGURE 2 1. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 1 MOD B

Sw RI Repo r t 03 .1 285 9.06 29 of 83FIGURE 2 2. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 2 MOD B

Sw RI Repo r t 03 .1 285 9.06 30 of 83FIGURE 2 3. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR NMHC METHOD 3 MOD B

Sw RI Repo r t 03 .1 285 9.06 31 of 83FIGURE 2 4. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 1 MOD B

Sw RI Repo r t 03 .1 285 9.06 32 of 83FIGURE 2 5. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 2 MOD B

Sw RI Repo r t 03 .1 285 9.06 33 of 83FIGURE 2 6. VALIDATION EDF PLOTS USING 23 REFERENCE NTE EVENTS FOR CO METHOD 3 MOD B

Sw RI Repo r t 03 .1 285 9.06 34 of 837.0 MONTE CARLO VALIDATION RESU LTS FROM 50 REFERENCE NTE EVENTS USING MOD 1 Based on the inform ation provided in the validation plots for the Mod 1, Mod 2, Mod 3 and Mod B runs, changes m ade to the vario u s steady-s t ate erro r su rfaces resulted in the validation of the MC model for all three em ission s and all three calculati on m e thods for each of the four Mod runs using the 23 ref e rence NTE events. EPA chose to continue the simu lation runs using only th e Mod 1 and Mod 2 revise d error surfaces by running an additional 2 7 reference N TE events (total = 50 reference N TE events) through the MC model. These two Mods were chosen because they both represen ted steady-s t ate NOx error surfaces that were level independen t (as com p ared to the M od 3 steady-state erro r s u rface that represen ted a com b ination of level dependent a nd level independent NO X error s). Th e resu lts f r o m these 50 simu lations were usedto calculate the m easurem ent allowances prov id ed in Se ctio n 9.0 of this report. This section deta ils the resu lts of the validation based on the 50 reference NTE events run with Mod 1. The Mod 1 simulations were perform e d at10,000 trials for NTE events with ideal BSNO X values less than 2.68204 g/kW -hr and at 30,00 trials otherwise. All 50 ref e rence NTE events m e t the convergence criteria listed in Table 5

. From the Mod 1 MC simulations, the validation 5 th and 95 th perc entile delta BS emissions were extracted from the output f iles for each of the 50 reference NTE events. Thes e percen tiles were then plotted as empirical distribution functions (EDF) to fo rm a valida tion inte rval f o r

th e on-road data. Also plotted was the EDF comp uted from the on-ro ad CE-CERT NTE events

. Figure 27 through Figure 29 repres e n t the va lida tion plo t s f o r the BSNO x for calculation m e thods 1, 2 and 3, respectively. Each of th e valid ation plots includes 5 th and 95 th percen tile E D Fs f o r the Mod 1 runs and the on-road CE-CERT data. Figure 30 through Figure 35 depict th e valid ation plots for the BSNM HC and BSCO simu lation runs, respectively. All BSNO X , B SNMH C and BSCO e m is sions validated for all three calculation m e thods fo r the Mod 1 runs with the 50 reference N TE events.

Sw RI Repo r t 03 .1 285 9.06 35 of 83FIGURE 2 7. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NO X METHOD 1 MOD 1

Sw RI Repo r t 03 .1 285 9.06 36 of 83FIGURE 2 8. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NO X METHOD 2 MOD 1

Sw RI Repo r t 03 .1 285 9.06 37 of 83FIGURE 2 9. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NO X METHOD 3 MOD 1

Sw RI Repo r t 03 .1 285 9.06 38 of 83FIGURE 3 0. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 1 MOD 1

Sw RI Repo r t 03 .1 285 9.06 39 of 83FIGURE 3 1. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 2 MOD 1

Sw RI Repo r t 03 .1 285 9.06 40 of 83FIGURE 3 2. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 3 MOD 1

Sw RI Repo r t 03 .1 285 9.06 41 of 83FIGURE 3 3. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 1 MOD 1

Sw RI Repo r t 03 .1 285 9.06 42 of 83FIGURE 3 4. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 2 MOD 1

Sw RI Repo r t 03 .1 285 9.06 43 of 83FIGURE 3 5. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 3 MOD 1

Sw RI Repo r t 03 .1 285 9.06 44 of 838.0 MONTE CARLO VALIDATION RESU LTS FROM 50 REFERENCE NTE EVENTS USING MOD 2 This section details the r esults of the valid ation based on the 50 reference NTE events run with Mod 2

. EPA chose to contin ue simu la tions using the Mod 1 an d Mod 2 revised error surfaces by running an additional 2 7 reference NTE events (to t al = 50 reference NTE events) through the MC m odel. These tw o Mods were chosen because they both represented steadystate NOx error surface s tha twere level indepe ndent (as comp ared to the Mod 3 steady-state error su rface that represented a co mbination of level dependent and level independent NO X errors). The Mod 2 simulations were perform e d at10,000 trials for NTE events with ideal BSNO X values less than 2.68204 g/kW -hr and at 30,00 trials otherwise. All 50 ref e rence NTE events m e t the convergence criteria listed in Table 5

. From the Mod 2 MC simulations, the validation 5 th and 95 th perc entile delta BS emissions were extracted from the output files for each of the 50 reference NTE events. These percen tile delta em issions were th en plotted as empirical di stribution functions (E DF ) to f o rm a va lida tion interval for the on-road data. Also plotted was the EDF computed from the on-ro ad CE-CERT NTE events. Figure 36 through Figure 38 represent the valida tion p l ots f o r the BSNO x for calculation m e thods 1, 2 and 3, respectively. Each of the validati on plots includes 5 th and 95 th percen tile E D Fs for the Mod 2 run s and the on-road CE-CERT data. Figure 39 through Figure 44 depict the validation plots for the BSNMHC and BSCO simu lation runs, respectively. All BSNO X , BSNMHC and BSCO e m is sions validated for all three calculation m e thods for the Mod 2 runs with the 50 reference NTE events.

Sw RI Repo r t 03 .1 285 9.06 45 of 83FIGURE 3 6. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NO X METHOD 1 MOD 2

Sw RI Repo r t 03 .1 285 9.06 46 of 83FIGURE 3 7. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NO X METHOD 2 MOD 2

Sw RI Repo r t 03 .1 285 9.06 47 of 83FIGURE 3 8. VALIDATION EDF PLOTS F O R 50 REF E RE NCE NTE EVE N TS FOR NO X METHOD 3 MOD 2

Sw RI Repo r t 03 .1 285 9.06 48 of 83FIGURE 3 9. VALIDATION EDF PLOTS F O R 50 REF E RE NCE NTE EVE N TS FOR NMHC METHOD 1 MOD 2

Sw RI Repo r t 03 .1 285 9.06 49 of 83FIGURE 4 0. VALIDATION EDF PLOTS F O R 50 REF E RE NCE NTE EVE N TS FOR NMHC METHOD 2 MOD 2

Sw RI Repo r t 03 .1 285 9.06 50 of 83FIGURE 4 1. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR NMHC METHOD 3 MOD 2

Sw RI Repo r t 03 .1 285 9.06 51 of 83FIGURE 4 2. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 1 MOD 2

Sw RI Repo r t 03 .1 285 9.06 52 of 83FIGURE 4 3. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 2 MOD 2

Sw RI Repo r t 03 .1 285 9.06 53 of 83FIGURE 4 4. VALIDATION EDF PLOTS USING 50 REFERENCE NTE EVENTS FOR CO METHOD 3 MOD 2

Table 7 pro v ides a sum m ary of all the va lida tio n resu lts f o r the var i ous MC simu lations as described in the above sections of this re port. All em issions and all calculation m e thods validated using the Mod 1 and Mod 2 runs with 50 reference NTE events while the Mod 3 and Mod B runs validated using 23 reference NTE events.

Sw RI Repo r t 03 .1 285 9.06 54 of 83TAB LE 7. SUMMARY OF VALI DATION RESUL T S Em ission Origina l 195 NTE Mod 1 50 NTE Mod 2 50 NTE Mod 3 23 NTE Mod B 23 NTE BSNO X Method 1 Validated Validated Validated Validated Validated BSNO X Method 2 Not Validated Validated Validated Validated Validated BSNO X Method 3 Not Validated Validated Validated Validated Validated BSNMHC Method 1 Validated Validated Validated Validated Validated BSNMHC Method 2 Validated Validated Validated Validated Validated BSNMHC Method 3 Validated Validated Validated Validated Validated BSCO Method 1 Not Validated Validated Validated Validated Validated BSCO Method 2 Not Validated Validated Validated Validated Validated BSCO Method 3 Not Validated Validated Validated Validated Validated

Sw RI Repo r t 03 .1 285 9.06 55 of 839.0 MEASURE M ENT AL LOWANCE CALCULATIONS As detailed in the original PEMS study, the m easurem ent error allowances were computed using both a regressi on m ethod and a m edian m ethod to determ ine the m easure m ent allowance. The proced ure was app lied to the s i mu lation d ata f o r the 5 0 ref erenc e NTE events from the Mod 1 and Mod 2 runs for each o f the three em issions and all three calcu lation m e thods. Figure 45 contains a regressi on plot of the 95 thperc entile delta BSNO x values (using Method 1 Mod 1) versus the ideal BSNO x values for the 50 reference NTE events. Included within the p l ot is the eq uation f o r th e f itted r egr ession lin e and the R-square (R 2 ) v a lue. Table 8 includes a comp arison of the results of the regression m e thod based on Figure 45 and the m edian m e thod (described in Ref 1). Under the heading “Regression Method” in the table, it is shown that the R-square criterion is not met by the data (R-square mu st be > 0.90). Thus, the m e dian m e thod mus tbe used. Under the heading “Media n Method” in the table, the m easurem ent error at the BSNO x threshold, based on using the m e dian of t he fifty 95 th percen tile delta BSNO x values, is 11.5932% when expressed as a per cent of the threshold value of 2.68204 g/kW -hr. Sim ilar regression plots and m easurem ent erro r tables are provided in the rem aining part of this section for the Mod 1 and Mod 2 re sults based on the 50 reference NTE events. Figure 45 through Figure 47 and Table 8 through Table 10 provide results for BSNO x Mod 1. Figure 48 through Figure 50 and Table 11 through Table 13 provide results for BSNMHC Mod 1, and Figure 51 through Figure 53 and Ta ble 14 through Table 16 provide results for BSCO Mod 1. Mod 2 results for BSNO x can be found in Figure 54 through Figure 56 and Table 17 through Table 19 , BSNMHC results ar e sho w n in Figure 57 through Figure 59 and Table 20 through Table 22 an d BSCO results are sho w n in Figure 60 through Figure 62 and Table 23 through Table 25

.

Sw RI Repo r t 03 .1 285 9.06 56 of 83N O x g/ k W - hr M e t hod 1 M od 1 5 0 R e f N T E Ev e n t s W i t h T i m e A lig n m e n t A d ju s t me n t

y = 0. 1213x +0. 0387 R

2 = 0. 8487

0.0 0 0.1 0 0.2 0 0.3 0 0.4 0 0.5 0 0.6 0 0.7 0 0.8 0 0.9 0

0 1 2 3 4 5 6 7 I d ea l N Ox g / kW - h r95th%DeltaNOx g/kW-hr

FIGURE 4 5. REGRE SSION PL OT OF 95TH PERCE NTILE DE LTA BSNO X VE RSUS IDEAL BS NO X FOR METHOD 1 MOD 1

TAB LE 8. MEASURE M ENT E R ROR AT T H RESHOLD FOR BS NO X USING REGRES SION AND M E DIAN ME THODS FOR METHOD 1 MOD 1 Regression Method Median Method
R 2 0.8487 Did Not Meet Crite ria RMSE(SEE) 0.0785 Met Crite ria 5% Median Ideal 0.1314 Predic ted 95 th% Delta at Threshold 0.3640 Median 95th% Delta 0.3109 Measurem ent Error @

Threshold=2.68204 13.5729% Measurem ent Error @ Threshold=2.68204 11.5932% 8

Sw RI Repo r t 03 .1 285 9.06 57 of 83N O x g/ k W - hr M e t hod 2 M od 1 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y = 0. 0558 x + 0. 0893 R

2 = 0.9058

0. 00 0. 05 0. 10 0. 15 0. 20 0. 25 0. 30 0. 35 0. 40 0. 45 0. 50

0 1 2 3 4 5 6 7 8 I d ea l N O x g / kW- h r95th%DeltaNOxg/kW-hr

FIGURE 4 6. REGRES SION PLOT OF 95TH PERCENT I LE DE LT A BSNO X VERS US IDEAL BS NO X FOR METHOD 2 MOD 1

TAB LE 9. MEASURE M ENT E R ROR AT T H RESHOLD FOR BS NO X USING REGRES SION AND M E DIAN ME THODS FOR METHOD 2 MOD 1 Regression Method Median Method R 2 0.9058 Met Crite ria RMSE(SEE) 0.0276 Met Crite ria 5% Median Ideal 0.1314 Predic ted 95 th% Delta at Threshold 0.2390 Median 95th% Delta 0.2270 Measurem ent Error @

Threshold=2.68204 8.9096% Measurem ent Error @ Threshold=2.68204 8.4641%

Sw RI Repo r t 03 .1 285 9.06 58 of 83N O x g/ k W - h r M e t h od 3 M o d 1 5 0 Re f N T E Ev e n ts W i th T i m e Al i g n m e n t A d j u s tm e n t

y =0.0869 x + 0

. 05 33 R

2 = 0.8714

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70

0 1 2 3 4 5 6 7 8 Id e a l N O x g / k W - h r95th%Delta NOxg/kW-hr

FIGURE 4 7. REGRE SSION PL OT OF 95TH PERCE NTILE DE LTA BSNO X VE RSUS IDEAL BS NO X FOR METHOD 3 MOD 1

TAB LE 10. MEASUREMENT E RROR AT THRESHOLD FOR B SNO X USI N G REGRES SION AND M E DIAN ME THODS FOR METHOD 3 MOD 1 Regression Method Median Method
R 2 0.8714 Did Not Meet Crite ria RMSE(SEE) 0.0512 Met Crite ria 5% Median Ideal 0.1314 Predic ted 95 th% Delta at Threshold 0.3000 Median 95th% Delta 0.2572 Measurem ent Error @

Threshold=2.68204 11.1845% Measurem ent Error @ Threshold=2.68204 9.5910%

Sw RI Repo r t 03 .1 285 9.06 59 of 83N M H C g/ k W - hr M e t hod 1 M od 1 5 0 R e f N T E Ev e n t s

y =0.0955x + 0. 0174 R

2 = 0.8109

0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07

0. 00 0. 05 0. 10 0.15 0.20 0. 25 0. 30 0. 35 0. 40 0. 45 I d ea l N M H C g / kW- h r95th%DeltaNMHC g/kW-hr

FIGURE 48. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC VERS US I D EAL BS NMHC FOR METHOD 1 MOD 1

TAB LE 11. MEASUREMENT E RROR AT THRESHOLD FOR B S NMHC USING REGRES SION AND M E DIAN ME THODS FOR METHOD 1 MOD 1 Regression Method Median Method
R 2 0.8109 Did Not Meet Crite ria RMSE(SEE) 0.0037 Did Not Meet Crite ria 5% Median Ideal 0.0002 Predic ted 95 th% Delta at Threshold 0.0443 Median 95th% Delta 0.0197 Measurem ent Error @

Threshold=0.28161 15.7288% Measurem ent Error @ Threshold=0.28161 7.0022%

Sw RI Repo r t 03 .1 285 9.06 60 of 83N M H C g/ k W - hr M e t hod 2 M od 1 5 0 R e f N T E Ev e n t s

y =0. 0382x + 0.0174 R

2 = 0. 4768

0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045

0. 00 0. 05 0.10 0.15 0. 20 0. 25 0.30 0.35 0. 40 0. 45 I d ea l N M H C g / kW- h r95th%DeltaNMHC g/kW-hr

FIGURE 49. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC VERS US I D EAL BS NMHC FOR METHOD 2 MOD 1

TAB LE 12. MEASUREMENT E RROR AT THRESHOLD FOR B S NMHC USING REGRES SION AND M E DIAN ME THODS FOR METHOD 2 MOD 1 Regression Method Median Method
R 2 0.4768 Did Not Meet Crite ria RMSE(SEE) 0.0032 Did Not Meet Crite ria 5% Median Ideal 0.0002 Predic ted 95 th% Delta at Threshold 0.0282 Median 95th% Delta 0.0187 Measurem ent Error @

Threshold=0.28161 9.9988% Measurem ent Error @ Threshold=0.28161 6.6274%

Sw RI Repo r t 03 .1 285 9.06 61 of 83N M H Cg/ k W - hr M e t hod 3 M od 1 5 0 R e f N T E Ev e n t s

y =0. 0545x + 0.0171 R

2 = 0. 6202

0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050

0. 00 0. 05 0.10 0.15 0.20 0. 25 0. 30 0. 35 0.40 0.45 I d ea l N M H C g / kW- h r95th%DeltaNMHC g/kW-hr

FIGURE 50. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC VERS US I D EAL BS NMHC FOR METHOD 3 MOD 1

TAB LE 13. MEASUREMENT E RROR AT THRESHOLD FOR B S NMHC USING REGRES SION AND M E DIAN ME THODS FOR METHOD 3 MOD 1 Regression Method Median Method
R 2 0.6202 Did Not Meet Crite ria RMSE(SEE) 0.0034 Did Not Meet Crite ria 5% Median Ideal 0.0002 Predic ted 95 th% Delta at Threshold 0.0324 Median 95th% Delta 0.0188 Measurem ent Error @

Threshold=0.28161 11.5222% Measurem ent Error @ Threshold=0.28161 6.6826%

Sw RI Repo r t 03 .1 285 9.06 62 of 83C O g/ k W - hr M e t hod 1 M od 1 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0. 1112x + 0.3148 R

2 = 0. 6518

0.00 0.20 0.40 0.60 0.80 1.00 1.20

0 1 2 3 4 5 6 7 I d ea l C O g / kW- h r95th%Delta COg/kW-hr

FIGURE 51. REGRESSION PLOT OF 95TH PERCE NTILE DE LTA BSCO VERS US IDEAL BS CO FOR METHOD 1 MOD 1

TAB LE 14. MEASUREMENT E RROR AT THRESHOLD FOR B S CO USING REGRES SION AND M E DIAN ME THODS FOR METHOD 1 MOD 1 Regression Method Median Method
R 2 0.6518 Did Not Meet Crite ria RMSE(SEE) 0.0846 Did Not Meet Crite ria 5% Median Ideal 0.0249 Predic ted 95 th% Delta at Threshold 0.9814 Median 95th% Delta 0.3600 Measurem ent Error @

Threshold=26.015 3.7726% Measurem ent Error @ Threshold=26.015 1.3840%

Sw RI Repo r t 03 .1 285 9.06 63 of 83C O g/ k W - hr M e t hod 2 M od 1 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0. 0543x + 0.3211 R

2 = 0. 3471

0.30 0.40 0.50 0.60 0.70 0.80 0.90

0 1 2 3 4 5 6 7 I d ea l C O g / kW- h r95th%Delta COg/kW-hr

FIGURE 52. REGRESSION PLOT OF 95TH PERCE NTILE DE LTA BSCO VERS US IDEAL BS CO FOR METHOD 2 MOD 1

TAB LE 15. MEASUREMENT E RROR AT THRESHOLD FOR B S CO USING REGRES SION AND M E DIAN ME THODS FOR METHOD 2 MOD 1 Regression Method Median Method
R 2 0.3471 Did Not Meet Crite ria RMSE(SEE) 0.0775 Did Not Meet Crite ria 5% Median Ideal 0.0249 Predic ted 95 th% Delta at Threshold 0.6466 Median 95th% Delta 0.3451 Measurem ent Error @

Threshold=26.015 2.4856% Measurem ent Error @ Threshold=26.015 1.3265%

Sw RI Repo r t 03 .1 285 9.06 64 of 83C O g/ k W - hr M e t hod 3 M od 1 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0.0988x + 0. 311 R

2 = 0. 5987

0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1.10

0 1 2 3 4 5 6 7 I d ea l C O g / kW- h r95th%Delta COg/kW-hr

FIGURE 53. REGRESSION PLOT OF 95TH PERCE NTILE DE LTA BSCO VERS US IDEAL BS CO FOR METHOD 3 MOD 1

TAB LE 16. MEASUREMENT E RROR AT THRESHOLD FOR B S CO USING REGRES SION AND M E DIAN ME THODS FOR METHOD 3 MOD 1 Regression Method Median Method
R 2 0.5987 Did Not Meet Crite ria RMSE(SEE) 0.0846 Did Not Meet Crite ria 5% Median Ideal 0.0249 Predic ted 95 th% Delta at Threshold 0.9033 Median 95th% Delta 0.3621 Measurem ent Error @

Threshold=26.015 3.4722% Measurem ent Error @ Threshold=26.015 1.3919%

Sw RI Repo r t 03 .1 285 9.06 65 of 83N O x g/ k W - hr M e t hod 1 M od 2 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0. 1272x - 0.0086 R

2 = 0.8638

-0

. 1 0 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

0 1 2 3 4 5 6 7
Id e a l N Ox g / k W- h r95th%Delta NOx g/kW-hr

FIGURE 5 4. REGRE SSION PL OT OF 95TH PERCE NTILE DE LTA BSNO X VE RSUS IDEAL BS NO X FOR METHOD 1 MOD 2

TAB LE 17. MEASUREMENT E RROR AT THRESHOLD FOR B SNO X USI N G REGRES SION AND M E DIAN ME THODS FOR METHOD 1 MOD 2 Regression Method Median Method
R 2 0.8638 Did Not Meet Crite ria RMSE(SEE) 0.0774 Met Crite ria 5% Median Ideal 0.1314 Predic ted 95 th% Delta at Threshold 0.3326 Median 95th% Delta 0.2834 Measurem ent Error @

Threshold=2.68204 12.3993% Measurem ent Error @ Threshold=2.68204 10.5681% 8

Sw RI Repo r t 03 .1 285 9.06 66 of 83N O x g/ k W - hr M e t hod 2 M od 2 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0.0614x + 0. 0375 R

2 = 0.9543

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50

0 1 2 3 4 5 6 7 Id e a l N O xg /k W- h r95th%Delta NOx g/kW-hr

FIGURE 5 5. REGRE SSION PL OT OF 95TH PERCE NTILE DE LTA BSNO X VE RSUS IDEAL BS NO X FOR METHOD 2 MOD 2

TAB LE 18. MEASUREMENT E RROR AT THRESHOLD FOR B SNO X USI N G REGRES SION AND M E DIAN ME THODS FOR METHOD 2 MOD 2 Regression Method Median Method R 2 0.9543 Met Crite ria RMSE(SEE) 0.0206 Met Crite ria 5% Median Ideal 0.1314 Predic ted 95 th% Delta at Threshold 0.2022 Median 95th% Delta 0.1935 Measurem ent Error @

Threshold=2.68204 7.5382% Measurem ent Error @ Threshold=2.68204 7.2141% 8

Sw RI Repo r t 03 .1 285 9.06 67 of 83N O x g/ k W - hr M e t hod 3 M od 2 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0.0932x + 0. 0056 R

2 = 0.8998

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70

0 1 2 3 4 5 6 7 Id e a l N Ox g / k W- h r95th%Delta NOx g/kW-hr

FIGURE 5 6. REGRE SSION PL OT OF 95TH PERCE NTILE DE LTA BSNO X VE RSUS IDEAL BS NO X FOR METHOD 3 MOD 2

TAB LE 19. MEASUREMENT E RROR AT THRESHOLD FOR B SNO X USI N G REGRES SION AND M E DIAN ME THODS FOR METHOD 3 MOD 2 Regression Method Median Method
R 2 0.8998 Did Not Meet Crite ria RMSE(SEE) 0.0477 Met Crite ria 5% Median Ideal 0.1314 Predic ted 95 th% Delta at Threshold 0.2556 Median 95th% Delta 0.2295 Measurem ent Error @

Threshold=2.68204 9.5288% Measurem ent Error @ Threshold=2.68204 8.5551% 8

Sw RI Repo r t 03 .1 285 9.06 68 of 83N M H C g/ k W - hr M e t hod 1 M od 2 5 0 r e f N T E Ev e n t s

y =0. 0951x + 0.0173 R

2 = 0.816

0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07

0. 00 0. 05 0. 10 0.15 0.20 0. 25 0. 30 0. 35 0. 40 0. 45 I d ea l N M H C g / kW- h r95th%DeltaNMHC g/kW-hr

FIGURE 57. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC VERS US I D EAL BS NMHC FOR METHOD 1 MOD 2

TAB LE 20. MEASUREMENT E RROR AT THRESHOLD FOR B S NMHC USING REGRES SION AND M E DIAN ME THODS FOR METHOD 1 MOD 2 Regression Method Median Method
R 2 0.8160 Did Not Meet Crite ria RMSE(SEE) 0.0036 Did Not Meet Crite ria 5% Median Ideal 0.0002 Predic ted 95 th% Delta at Threshold 0.0441 Median 95th% Delta 0.0194 Measurem ent Error @

Threshold=0.28161 15.6533% Measurem ent Error @ Threshold=0.28161 6.8769%

Sw RI Repo r t 03 .1 285 9.06 69 of 83N M H C g/ k W - hr M e t hod 2 M od 2 5 0 R e f N T E Ev e n t s

y =0.0386x + 0. 0172 R

2 = 0. 479

0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045

0. 00 0. 05 0.10 0.15 0. 20 0. 25 0.30 0.35 0. 40 0. 45 I d ea l N M H C g / kW- h r95th%DeltaNMHC g/kW-hr

FIGURE 58. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC VERS US I D EAL BS NMHC FOR METHOD 2 MOD 2

TAB LE 21. MEASUREMENT E RROR AT THRESHOLD FOR B S NMHC USING REGRES SION AND M E DIAN ME THODS FOR METHOD 2 MOD 2 Regression Method Median Method
R 2 0.4790 Did Not Meet Crite ria RMSE(SEE) 0.0032 Did Not Meet Crite ria 5% Median Ideal 0.0002 Predic ted 95 th% Delta at Threshold 0.0281 Median 95th% Delta 0.0182 Measurem ent Error @

Threshold=0.28161 12.3993% Measurem ent Error @ Threshold=0.28161 6.4499%

Sw RI Repo r t 03 .1 285 9.06 70 of 83N M H Cg/ k W - hr M e t hod 3 M od 2 5 0 R e f N T E Ev e n t s

y =0.054x + 0. 017 R

2 = 0.6195

0.010 0.015 0.020 0.025 0.030 0.035 0.040

0. 00 0. 05 0.10 0.15 0.20 0. 25 0. 30 0. 35 0.40 0.45 I d ea l N M H C g / kW- h r95th%DeltaNMHC g/kW-hr

FIGURE 59. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSNMHC VERS US I D EAL BS NMHC FOR METHOD 3 MOD 2

TAB LE 22. MEASUREMENT E RROR AT THRESHOLD FOR B S NMHC USING REGRES SION AND M E DIAN ME THODS FOR METHOD 3 MOD 2 Regression Method Median Method
R 2 0.6195 Did Not Meet Crite ria RMSE(SEE) 0.0034 Did Not Meet Crite ria 5% Median Ideal 0.0002 Predic ted 95 th% Delta at Threshold 0.0322 Median 95th% Delta 0.0182 Measurem ent Error @

Threshold=0.28161 11.4367% Measurem ent Error @ Threshold=0.28161 6.4801%

Sw RI Repo r t 03 .1 285 9.06 71 of 83C O g/ k W - hr M e t hod 1 M od 2 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0. 1156x + 0.3087 R

2 = 0. 6865

0.00 0.20 0.40 0.60 0.80 1.00 1.20

0 1 2 3 4 5 6 7 I d ea l C O g / kW- h r95th%Delta COg/kW-hr

FIGURE 60. REGRESSION PLOT OF 95TH PERCE NTILE DE LTA BSCO VERS US IDEAL BS CO FOR METHOD 1 MOD 2

TAB LE 23. MEASUREMENT E RROR AT THRESHOLD FOR B S CO USING REGRES SION AND M E DIAN ME THODS FOR METHOD 1 MOD 2 Regression Method Median Method
R 2 0.6865 Did Not Meet Crite ria RMSE(SEE) 0.0813 Did Not Meet Crite ria 5% Median Ideal 0.0249 Predic ted 95 th% Delta at Threshold 1.0017 Median 95th% Delta 0.3576 Measurem ent Error @

Threshold=26.015 3.8505% Measurem ent Error @ Threshold=26.015 1.3746%

Sw RI Repo r t 03 .1 285 9.06 72 of 83C O g/ k W - hr M e t hod 2 M od 2 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0. 0567x + 0.3162 R

2 = 0. 3785

0.30 0.40 0.50 0.60 0.70 0.80 0.90

0 1 2 3 4 5 6 7 I d ea l C O g / kW- h r95th%Delta COg/kW-hr

FIGURE 61. REGRESSION PLOT OF 95TH PERCE NTILE DE LTA BSCO VERS US IDEAL BS CO FOR METHOD 2 MOD 2

TAB LE 24. MEASUREMENT E RROR AT THRESHOLD FOR B S CO USING REGRES SION AND M E DIAN ME THODS FOR METHOD 2 MOD 2 Regression Method Median Method
R 2 0.3785 Did Not Meet Crite ria RMSE(SEE) 0.0756 Did Not Meet Crite ria 5% Median Ideal 0.0249 Predic ted 95 th% Delta at Threshold 0.6561 Median 95th% Delta 0.3361 Measurem ent Error @

Threshold=26.015 2.5221% Measurem ent Error @ Threshold=26.015 1.2919%

Sw RI Repo r t 03 .1 285 9.06 73 of 83C O g/ k W - hr M e t hod 3 M od 2 5 0 R e f N T E Ev e n t s W i t h T i me A lig n m e n t A d ju s t me n t

y =0. 1009x + 0.3065 R

2 = 0. 6195

0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1.10

0 1 2 3 4 5 6 7 I d ea l C O g / kW- h r95th%Delta COg/kW-hr

FIGURE 62. REGRESSION PLOT OF 95TH PERCE NTILE DE LTA BSCO VERS US IDEAL BS CO FOR METHOD 3 MOD 2

TAB LE 25. MEASUREMENT E RROR AT THRESHOLD FOR B S CO USING REGRES SION AND M E DIAN ME THODS FOR METHOD 3 MOD 2 Regression Method Median Method
R 2 0.6195 Did Not Meet Crite ria RMSE(SEE) 0.0827 Did Not Meet Crite ria 5% Median Ideal 0.0249 Predic ted 95 th% Delta at Threshold 0.9114 Median 95th% Delta 0.3513 Measurem ent Error @

Threshold=26.015 3.5033% Measurem ent Error @ Threshold=26.015 1.3505%

Sw RI Repo r t 03 .1 285 9.06 74 of 83Although only 23 reference NTE events we re simu lated under the Mod 3 and Mod B runs, the measurem ent errors were com puted for all three em issions and calculation m ethods using the sam e regression and m e dian m e thods. Table 26 lists a summary of the measurem ent errors comp uted from simulation results in this study along with the resu lts from the original PEMS study. Note that BSNO x for m e thods 2 and 3 and BSCO for all three m e thods did not validate in the original study.
TAB LE 26. SUMMARY OF MEASUREM E NT ERRORS AT RESPE C TIVE THRESHOLD ( % ) Em ission # Ref NTE Events Model Run Calculation Method Method 1 Method 2 Method 3 BSNO X 195 Original 22.30 4.45 * 6.61 * 50 Mod 1 11.59 8.91 9.59 50 Mod 2 10.56 7.54 8.56 23 Mod 3 11.78 8.01 10.78 23 Mod B 11.93 8.26 10.33 BSNMHC 195 Original 10.08 8.03 8.44 50 Mod 1 7.00 6.63 6.68 50 Mod 2 6.88 6.45 6.48 23 Mod 3 7.28 6.90 6.87 23 Mod B 7.25 6.82 6.78 BSCO 195 Original 2.58 * 1.99 * 2.11 * 50 Mod 1 1.38 1.33 1.39 50 Mod 2 1.37 1.29 1.35 23 Mod 3 1.52 1.41 1.48 23 Mod B 2.19 1.99 2.16

* Methods did not validate The f i nal B S m easurement a llowan ces were co mputed by multiply ing each of th e three m easure m ent errors (in pe rcent) tim es their corre sponding threshold values. Table 27 lis ts the m easure m ent allowances for all simulation M od runs and the original PEMS study. The em issions thresholds (g/hp-hr) are provided in the first column of Table 27

.

Sw RI Repo r t 03 .1 285 9.06 75 of 83TABLE 27. SUMMARY OF MEASUREMENT ALLOWANCE, G/HP-HR Em ission # Ref NTE Events Model Run Calculation Method Method 1 Method 2 Method 3 BSNOx Threshold=2.0 195 Original 0.446 0.089 * 0.132 * 50 Mod 1 0.232 0.178 0.192 50 Mod 2 0.211 0.151 0.171 23 Mod 3 0.236 0.160 0.216 23 Mod B 0.239 0.165 0.207 BSNMHC Threshold=0.21 195 Original 0.021 0.017 0.018 50 Mod 1 0.015 0.014 0.014 50 Mod 2 0.014 0.014 0.014 23 Mod 3 0.015 0.014 0.014 23 Mod B 0.015 0.014 0.014 BSCO Threshold=19.40 195 Original 0.501 * 0.386 * 0.409 * 50 Mod 1 0.268 0.258 0.270 50 Mod 2 0.266 0.250 0.262 23 Mod 3 0.295 0.274 0.287 23 Mod B 0.425 0.386 0.419

* Methods did not validate

Sw RI Repo r t 03 .1 285 9.06 76 of 8310.0 VALI DATI O N SENSI T IVITY RESUL T S F R OM 13 REFERENCE NT E EVENTS Simu lation results for Mod 1, Mod 2, Mod 3 and Mod B from the 13 reference NTE events cho s en in the original s t udy produced sens itiv ity v alues from the valid ation runs due to varian ce an d bias for all th ree 95 th percentile delta em issions by al l three calculation m e thods. The tables below summ arize the error su rf ac es in whicheithe r the contr i bution

- to-var iance norm alized sensitivity average valu e or the ‘on/off’ bias check for the error su rface was at least 5% in absolute m agnitude com p ared to all the other erro r surfaces. If the label in the error surface contains the wo rds ‘OnOff’ then it rep r es ents a ch eck for bias ; otherwise, the error surface ind i cates a check for varian ce. If at least 5 of the 1 3 reference NTE events resulted in a sensitivity average valu e > 5% or < -5% then th e average co ntribu tion to varianc e is included in the tab l es. Table 28 lis ts the BSNO X sensitivity erro r surfaces assoc i a t ed with the origina l PEMS study for Mod B, Mod 1, Mod 2 and the Mod 3 M onte Carlo validation simulations perform e d during this projec t. Note in th e output results that the bias due to steady-state exhaust flow rat e in Meth od 1 in th e or iginal stud y ha s been e lim inated in the f our m odif i cation simu lations. In addition, the steady-state CO 2 bias has been elim inated, but a sensitivity com p onent due to variance has been added for Methods 2 and 3. Table 29 lis ts the s e nsitivities due to BSNMHC. Note that no changes have occurred for the four m o difications relative to the original study. This is due to the fact that the revised error surfaces didnot effect the NMHC calculation s. Table 30 summ arizes the resu lts of the BSCO sensitivitie s wherein the steady- st ate CO bias found in the origin al m odeling has been elim inated in Mod 1, M od 2 and Mod 3 and replaced by an increas ed sen sitiv ity du e to varian ce.

Sw RI Repo r t 03 .1 285 9.06 77 of 83TAB LE 28. SENSITI V ITY RES U LTS COMPARING 13 REFE RENCE NT E EVE N TS ACROSS 5 MONTE CARLO VALIDAT ION SIMULAT I ONS FOR BSNO X Method # Error Surfac e Avg Contribution to Norm alized Variance, % Mod

B Mod 1 Mod 2 Mod 3

1 1 SSNOx 29.0 33.2 42.9 32.0 36.3 20 SS Exhaust Flow 9.6 19.2 16.6 20.4 18.8 51 NOx Ti m e Alignm ent 9.0 10.7 9.1 10.9 10.6 Pulse_Flow_OnOff 25.8 30.5 26.3 33.1 29.1 SS Exhaust Flow_OnOff 20.4

2 1 SSNOx 37.0 46.9 57.8 44.6 52.5 2 TRNOx 5.5 6.6 45 SSCO2 -40.1 -33.7 -43.6 -37.7 51 NOx Ti m e Alignm ent 6.7 7.5 7.6 6.2 SSCO2_OnOff -54.3

3 1 SSNOx 35.6 44.0 54.9 41.6 49.4 45 SSCO2 -37.0 -31.3 -39.6 -34.8 51 NOx Ti m e Alignm ent 10.4 13.9 11.5 15.8 12.9 SSCO2_OnOff -50.9 DEFINITIONS Original April 2007 Study: A ll origin al erro r surfaces Mod B Revised SS Exhaust Flo w and SS CO2 error su rfaces Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error surfaces Origina l

Sw RI Repo r t 03 .1 285 9.06 78 of 83TAB LE 29. SENSITI V ITY RES U LTS COMPARING 13 REFE RENCE NT E EVE N TS ACROSS 5 MONTE CARLO VALIDATIO N SIMULAT I ONS FOR BSNMHC
Method # Error Surfac e Avg Contribution to Norm alized Variance, % Mod

B Mod 1 Mod 2 Mod 3 1 13 SSNMHC 7.1 7.1 7.0 6.9 6.6 19 NMHC Ambient 60.9 62.8 63.2 62.5 63.2 SS NMHC_OnOff 14.5 14.3 13.5 12.9 14.2

2 13 SSNMHC 6.7 6.8 6.8 6.9 6.4 19 NMHC Ambient 64.5 66.6 66.7 66.3 66.9 SS NMHC_OnOff 14.3 12.9 13.5 13.2 12.8

3 13 SSNMHC 6.7 6.9 6.8 7.0 6.4 19 NMHC Ambient 64.4 66.5 66.6 66.1 66.8 SS NMHC_OnOff 14.4 13.0 13.6 12.5 13.0 DEFINITIONS Original April 2007 Study: A ll origin al erro r surfaces Mod B Revised SS Exhaust Flo w and SS CO2 error su rfaces Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error surfaces Origina l

Sw RI Repo r t 03 .1 285 9.06 79 of 83TABLE 30. SENSITI V ITY RESULTS COM P ARING 1 3 REFE RE NCE NTE EVE N TS ACROSS 5 MONTE CARLO VALIDATIO N SIMULAT I ONS FOR BSCO
Method # Error Surfac e Avg Contribution to Norm alized Variance, % Mod

B Mod 1 Mod 2 Mod 3
1 7 SSCO 6.2 6.5 79.3 79.1 79.3 52 CO Ti m e Alignm ent 10.3 11.0 10.1 10.7 SSCO_OnOff 82.0 82.2

2 7 SSCO 6.4 6.7 82.9 83.0 83.0 52 CO Ti m e Alignm ent 10.5 11.8 10.4 SSCO_OnOff 83.6 84.6

3 7 SSCO 6.2 6.3 77.9 78.0 77.9 52 CO Ti m e Alignm ent 11.8 10.1 19.2 17.9 19.0 SSCO_OnOff 80.8 81.0 DEFINITIONS Original April 2007 Study: A ll origin al erro r surfaces Mod B Revised SS Exhaust Flo w and SS CO2 error su rfaces Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error surfaces Origina l

Sw RI Repo r t 03 .1 285 9.06 80 of 8311.0 FULL MO DEL S E NS ITIVITY RESULTS F R OM 13 REFERENCE NTE EVENTS This section contains a summary of the erro r s u rfaces that contributedthe m o st to bias and variance of the BSNO X e m issi ons based on the full MC m odel. Simu lation results from t h e 13 reference NTE events produced se nsitivity values for all three 95 thper centile d elta em issions by all three calculation m e thods. Table 31 lists the e rror su rf ace s in whicheithe rthe contribution-to-variance norm a lized sensitivity average v a lue or the ‘o n/off’ bias c h eck for the error surface was at least 5% in abs olute m a gnitude compared to all the other error surfaces for BSNO X . If at least 5 of the 13 refe rence NTE events resultedin a sens itiv ity ave r age value > 5 % or < -5% then the averag e contribution to variance is included in the table. The sensitivity results from the original study are included for comp arison. The results from the BSNO X Method 1 sensitivities show that the steady-state exh a ust flow rate bias from the original study has been elim inated in Mod B, Mod 1, Mod 2, and Mod 3. There is als o a slight in creas e (from the origin al study) in th e average sensitiv ity due to variation in the stead y-sta t e exhaust f l ow rate with all the m odi fication runs. Slight increases in the average sens itiv ity f o r th e steady

- state NO X error surface are s een in all m odification runs except Mod 2. Method 2 results for BSNO X show that the ste ad y -sta te CO 2 bias effect and the NO X time alignm ent variation effect were elim inated in all the m odi fication runs. However, new error surfaces tha tshowed sensitiv ities in the m odi fication runs were the ste ady-sta t e C O 2 variation effect and the BSFC DOE Test bias effect. Th ere were also sligh tincreases in the averag e sensitivity due to variation for the steady-state N O X error surface in all th e m odification runs. The steady-s t ate CO 2 bias effect for Method 3 was elim in ated in all the modification runs and a variation effect due to steady

- s t ate CO 2 was included. Also, there wa s a slight incre ase in the averag e sensitivity due to variation for the steady

- s t ate NO X error surfac e in all the modification runs.

Sw RI Repo r t 03 .1 285 9.06 81 of 83TAB LE 31. SENSITI V ITY RES U LTS COMPARING 13 REFE RENCE NT E EVE N TS ACROSS 5 MONTE CARLO FULL MO DE L SIMULATIONS FOR BSNO X Method # Error Surfac e Avg Contribution to Norm alized Variance, % Mod B Mod 1 Mod 2 Mod 3

1 1 SSNOx 21.0 22.3 31.2 20.8 25.4 20 SS Exhaust Flow 6.3 10.2 9.5 11.0 10.4 31 War m -up torque -23.2 -23.8 -20.7 -23.7 -22.6 35 Engine Manuf Torque -9.8 -9.9 -8.7 -9.9 -9.4 51 NOx Ti m e Alignm ent 6.1 5.5 5.0 5.5 5.6 Pulse_Flow_OnOff 14.7 15.1 13.8 16.4 14.8 SS Exhaust Flow_OnOff 14.6

2 1 SSNOx 29.5 33.3 43.9 30.4 37.1 38 War m -up BSFC 6.3 5.9 5.2 6.2 5.6 42 Engine Manuf BSFC 13.6 14.1 12.3 15.8 14.0 45 SSCO2 -23.2 -20.0 -24.1 -21.9 51 NOx Ti m e Alignm ent 6.1 BSFC DOE Test_OnOff -5.9 -5.3 -6.5 -6.1 SSCO2_OnOff -34.9

3 1 SSNOx 23.6 25.6 35.1 34.2 29.2 31 War m -up Torque -27.8 -28.5 -25.0 -29.4 -27.0 35 Engine Manuf Torque -11.7 -11.9 -10.4 -12.6 -11.8 45 SSCO2 -16.3 -14.8 -16.8 -15.6 51 NOx Ti m e Alignm ent 6.0 5.4 4.8 5.8 5.3 SSCO2_OnOff -25.2 DEFINITIONS Original April 2007 Study: A ll origin al erro r surfaces Mod B Revised SS Exhaust Flo w and SS CO2 error su rfaces Mod 1 Revised SSNOx Mod 1, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 2 Revised SSNOx Mod 2, SS Exhaust Flow, SSCO and SSCO2 error surfaces Mod 3 Revised SSNOx Mod 3, SS Exhaust Flow, SSCO and SSCO2 error surfaces Origina l

Sw RI Repo r t 03 .1 285 9.06 82 of 8312.0 REFE RE NCES

1. Feist, M.D., Sharp, C. A., Mason, R.L., a nd Buc k ingham , J.P., “Determ i nation of PEMS Measurem ent Allowances for Gaseous Em issions Regulated Under the Heavy-Duty Diesel Engine In-Use T e sting Program ,” SwRI Project 03-12024, U.S. Environm ental Protection Agency, California Air Resour ces Board and Engine Manufacturers Association, April 20.

Sw RI Repo r t 03 .1 285 9.06 83 of 83