PM-PEMS Measurement Allowance 

Determination

Draft Final Report

Prepared by:

Imad A. Khalek and Thomas L. Bougher

Robert L. Mason and Janet P. Buckingham

Southwest Research Institute

May 17, 2010

Prepared for:

U.S. Environmental Protection Agency

California Air Resources Board

Engine Manufacturers Association
ii

SwRI Project No. 14936.12

FORWARD

The PM-PEMS measurement allowance program was performed by the Department of 

Emissions Research and Development under Mr. Jeff White, Director. Dr. Imad Khalek, 

Program Manager, was the Principal Investigator and Project Manager, Mr. Thomas Bougher, 

Research Engineer, was the Project Leader, and Mr. Daniel Preece, Research Assistant, was the 

laboratory technical assistant. Dr. Bob Mason, Institute Analyst, was the Principal CoInvestigator responsible for statistical analysis, and Ms. Janet Buckingham, Staff Analyst, was 

the Project Leader/Statistics. Other SwRI Emissions R&D staff with contribution to the project 

were Mr. Michael Feist, Senior Research Engineer, Mr. Richard Mechler, Senior Research 

Technologist, Mr. Don Parker, Senior Technician, Mr. Jose Sosa, Principal Technician, Mr. 

Keith Echtle, Laboratory Assistant Manager, and Mr. Ernie Krueger, Laboratory Manager. 

Additional SwRI assistance during Environmental Testing was provided by Rick Pitman, Senior 

Engineering Technologist, Mr. Mike Negrete, Senior Technician, Mr. David Smith, Staff 

Technician, Mr. Herbert Walker, Senior Engineering Technologist, and Mr. Eric Dornes, 

Principal Engineer.

This work was performed under Environmental Protection Agency (EPA) Work Assignments 2-

7, 3-7, 4-7, and 2-12, under SwRI Project Numbers 12859.07, 13749.07, 14658.07, and 

14936.12, respectively. The original EPA Work Assignment Manager was Mr. Matt Spears. 

After March of 2009, the EPA Work Assignment Manager became Mr. Chris Laroo. This work 

started in June of 2007 and will end by June 26, 2010. Testing started in June of 2008 and ended 

in September of 2009. 

Funding for this work was provided by U.S EPA, Engine Manufacturers Association (EMA), and 

the California Air Resources Board (CARB). Funding by EMA and CARB was provided directly 

to EPA in support of this work. 

In-kind engine and technical support were provided by Volvo Powertrain. In-kind PM-PEMS 

and technical support were provided by AVL, Horiba, and Sensors. SwRI acknowledges the 

following individuals for their laboratory technical and logistical support:

Mr. Jeff Saxon and Mr. Steve Trevitz, from Volvo Powertrain

Mr. Scott Porter, Dr. Mike Akard and Dr. Qiang Wei, from Horiba

Mr. Bill Silvis, Mr. Siegfried Roeck, Dr. Wolfgang Schindler, Dr. Roland Wanker, Dr. Michael 

Arndt, Mr. P.J. Pankratz, and Ms. Sarah Kingham, from AVL

Mr. Andrew Reading, Dr. David Booker, Dr. Atul Shah, Mr. Carl Ensfield, Mr. Tim Bottomley, 

and Mr. Kevin Bouma, from Sensors
iii

SwRI Project No. 14936.12

A measurement allowance steering committee (SC) composed of EMA, EPA, CARB, and PMPEMS manufacturer members met on a regular basis throughout the entire project to discuss the 

progress made and make recommendations. The SC has contributed significantly to this project. 

SwRI acknowledges the following SC members for their active participation:

Mr. Rey Agama, Caterpillar

Dr. Mike Akard, Horiba

Dr. Dipak Bishnu, CARB

Dr. David Booker, Sensors

Dr. Bruce Cantrell, EPA Consultant

Mr. Tim French, EMA

Dr. Bob Giannelli, EPA

Mr. Denny Hao, PACCAR

Dr. Kent Johnson, CE-CERT

Mr. Craig Kazmierczak, Detroit Diesel

Mr. John Kegebein, John Deere

Mr. Tom Kramer, Navistar

Mr. Chris Laroo, EPA

Mr. Hector Maldonado, CARB

Mr. Bill Martin, Cummins

Dr. Shirish Shimpi, Cummins

Mr. Bill Silvis, AVL

Ms. Carole Smith, Isuzu

Mr. Matt Spears, EPA

Mr. Steve Trevitz, Volvo Powertrain

Dr. Qiang Wei, Horiba

 
iv

SwRI Project No. 14936.12

EXECUTIVE SUMMARY

The United States Environmental Protection Agency (EPA), Engine Manufacturers Association 

(EMA) and California Air Resources Board (CARB) agreed to pursue an experimental data 

driven program to establish a measurement allowance (MA) for in-use particulate matter (PM) 

testing using PM portable emissions measurement systems (PM-PEMS). The MA is a brakespecific PM emissions error associated with using in-use PM-PEMS equipment compared to the 

laboratory reference filter method. If the MA error is a positive value above zero, after EPA 

rounding method, it will increase the EPA in-use not-to-exceed (NTE) standard by the rounded 

value. If the error is negative or zero, it will not contribute to any changes to the in-use standard. 

The measurement allowance steering committee (SC) accepted the following PM-PEMS to be 

part of the MA program:

• Sensors Particle Proportional Micro Diluter (PPMD). A PM-PEMS that uses 

proportional dilution and a series of 8 quartz crystal microbalances to measure total 

(solid plus volatile) PM. The total PM is measured as a single flow-weighted value for a 

NTE event.

• Horiba Transient Particulate Matter (TRPM). A PM-PEMS that uses proportional 

dilution, a real time electrical aerosol detector, and an integrated filter sample to report 

total PM. The total PM is measured as a single flow-weighted value for a NTE event. 

This instrument can report real time total PM, but was not used as such on this program.

• AVL micro-soot sensor (MSS). An instrument that uses constant dilution and a photo 

acoustic detector to measure soot or the elemental carbon portion of PM. Soot is

measured in real time during a NTE event. 

The SC agreed that only the PPMD by Sensors and TRPM by Horiba will be used for the official 

determination and validation of the measurement allowance generated because both are 

designated as complete PEMS and both measure total (solid plus volatile) PM, as required by US 

EPA to be valid PM-PEMS. The SC also agreed that only the PM-PEMS that produces the 

lowest positive 95th

• Laboratory steady-state (SS) and transient engine NTE testing using PM-PEMS and 

CVS filter measurement during SS testing only. The SS was used to capture bias, 

compared to the CVS, and the transient was used to capture precision only since there is 

no reference method to measure PM during a short NTE event. A 2007 heavy-duty 

percentile measurement allowance, based on Sensors or Horiba only, will be 

chosen for in-use validation by CE-CERT due to funding limitation. The third instrument, the 

MSS by AVL, was not a complete PM-PEMS and it was always used in conjunction with the 

PPMD on this program, based on an agreement reached between Sensors and AVL. The SC 

agreed that the MSS will not be considered as an option for official measurement allowance 

determination, unless both the PPMD and the TRPM failed validation.

The PM-PEMS-MA project included four main elements:
v

SwRI Project No. 14936.12

diesel Mack MP7, by Volvo Powertrain, was used to conduct the engine experiments. 

The engine was configured with a bypass around the diesel particle filter (DPF) to 

provide PM concentration levels similar to those expected at the NTE threshold limits 

between 0.02 g/hp-hr and 0.03 g/hp-hr. 

• Environmental testing such as the effect of shock and vibration, pressure, temperature 

and relative humidity, and electric noise on PM-PEMS precision. 

• Monte Carlo simulation to determine a brake-specific measurement allowance value

using error surfaces generated from the PM-PEMS laboratory and environmental 

testing, and from error surfaces generated during the gaseous PEMS program

o A total of 141 Reference NTEs were provided by EMA as an input to the model 

for calculating ideal brake specific emissions, prior to the Monte Carlo 

simulation

• Model validation using data generated from in-use PM-PEMS testing by CE-CERT

Three methods are used to determine in-use NTE brake-specific PM emissions:

• Method 1 f(PPPP�����, torquei, speedi, exhaust-flowi

• Method 2 f(PPPP�����, torque

)

i, speedi, exhaust- and fuelECM, G- flowi

• Method 3 f(PM

) 

i, torquei, speedi, and fuelECM, G-flowi

All methods require engine control module (ECM) broadcasted torque and speed. In addition, 

Method 1 requires measured exhaust flow but not fuel flow; Method 2 requires measured exhaust 

flow, ECM broadcasted fuel flow, and gas-based fuel flow; Method 3 is similar to Method 2, but 

it does not require exhaust flow. Besides real time PM measurement, all methods can use a single 

flow-weighted PM measurement for a NTE, except Method 3, where real time PM measurement 

is required. Thus, Method 1 and Method 2 were applied to all three PM-PEMS, but Method 3 

was only applied to the AVL MSS.

Compared to Horiba TRPM, the Sensors PPMD, as shown in Table 1, produced the lowest 

positive 95

)

th percentile measurement allowance of 0.0061 g/hp-hr for a NTE threshold level of 

0.02 g/hp-hr, using calculation Method 2. The Horiba TRPM produced a measurement allowance 

value of 0.0100 g/hp-hr. Thus, the PPMD was selected by the SC for in-use testing by CE-CERT 

for Monte Carlo model validation. Although not accepted as a PM-PEMS, the AVL MSS 

produced the lowest measurement allowance of essentially zero. The SC agreed to include the

AVL MSS during in-use validation because it was used in conjunction with the PPMD during 

the laboratory portion of the testing.
vi

SwRI Project No. 14936.12

TABLE 1. MEASRUEMENT ALLOWANCE BASED ON 0.02 G/HP-HR THRESHOLD

Measurement Errors at NTE Threshold, g/hp-hr

PEMS Method 1 Method 2 Method 3

AVL 0.0001 -0.0005 -0.0005

Horiba 0.0109 0.0100 n/a

Sensors 0.0069 0.0061 n/a

Figures 1 and 2 show examples of validation plots for Sensors PPMD using Method 1 and AVL 

MSS using Method 3. The dots on each of the two figures represent the in-use delta BSPM 

between PEMS and CE-CERT on the y-axis versus CE-CERT BSPM on the x-axis (Ideal 

BSPM). Similarly, the lines represents the 5th and 95th percentile errors produced by MonteCarlo simulation based on laboratory testing versus a reference NTE BSPM value on the x-axis

(Ideal BSPM).To pass model validation, ≤ 10% of the in-use NTE delta brake-specific PM 

(BSPM) between the PEMS and the CE-CERT (dots on Figures 1 and 2) must reside outside the 

95th percentile and 5th percentile lines. 

The Sensors PPMD failed validation using Method 1 and Method 2, as shown in Table 2. For 

Method 1, 32 percent of the data were below the 5th percentile, and for Method 2, 34 percent of 

the data were below the 5th percentile. The failure was mainly due to negative bias. The exact 

cause for negative bias is unknown, but could be related to special environmental conditions 

experienced during in-use testing that the Sensors PPMD did not account for properly.

The AVL MSS passed validation using Method 3, as shown in Table 3, where less than 10 

percent of the data were outside the 5th and 95th percentiles, with the majority of the failed points 

being above the 95th percentile. Method 2 failed validation by one percentage point, and Method 

1 failed validation by having18 percent of the data outside the 5th and 95th percentiles, with the 

majority of failed points being above the 95th percentile. 
vii

SwRI Project No. 14936.12

FIGURE 1. IN-USE VALIDATION PLOT FOR SENSORS PPMD USING METHOD 1. 

(Y AXIS IS THE DIFFERENCE BETWEEN PEMS AND IDEAL BSPM. IDEAL BSPM 

IS THE LABORATORY CVS BSPM FOR THE 5TH AND 95TH AND THE CE-CERT 

BSPM FOR THE DOTS)
viii

SwRI Project No. 14936.12

FIGURE 2. IN-USE VALIDATION PLOT FOR AVL MSS USING METHOD 3. (Y AXIS 

IS THE DIFFERENCE BETWEEN PEMS AND IDEAL BSPM. IDEAL BSPM IS THE 

LABORATORY CVS BSPM FOR THE 5TH AND 95TH AND THE CE-CERT BSPM FOR 

THE DOTS)
ix

SwRI Project No. 14936.12

TABLE 2. SENSORS PPMD VALIDATION RESULTS

Method 1 Method 2

Total # CE-CERT Points 217 217

# CE-CERT Points 

within Ideal BSPM 

Range

210 210

# CE-CERT Points above 

95th Percentile 0 1

# CE-CERT Points below 

5th Percentile 68 70

# CE-CERT Points 

between 5th and 95th 

Percentiles

142 139

% CE-CERT points that 

did not validate 32.38 33.81

TABLE 3. AVL MSS VALIDATION RESULTS

Method 1 Method 2 Method 3

Total # CE-CERT Points 271 271 271

# CE-CERT Points 

within Ideal BSPM 

Range

263 263 263

# CE-CERT Points above 

95th Percentile 47 28 23

# CE-CERT Points below 

5th Percentile 2 2 3

# CE-CERT Points 

between 5th and 95th 

Percentiles

222 233 237

% CE-CERT points that 

did not validate 18.08 11.41 9.89
x

SwRI Project No. 14936.12

The measurement allowance program was concluded by the SC because one of the instruments,

namely the AVL MSS, passed its own validation criteria using Method 3. However, the MSS as 

used in this program is not accepted as a PM-PEMS by EPA, and the measurement allowance 

generated based on the performance of this instrument is not an official part of the measurement 

allowance.

The final official measurement allowance accepted by the SC was based on the Sensors PPMD. 

Further investigation of why the PPMD did not validate the model is being investigated outside 

this program, and the details are expected to be part of the CE-CERT Final Report on in-use 

validation testing. As for the status of the Horiba TRPM, it is understood that EPA may consider 

approving this system, although it did not go through validation. As for the status of the AVL 

MSS, it is understood that EPA may consider approving this system along with a required filter 

box to collect total PM to span the real time soot signal produced by the photo acoustic sensor. 
xi

SwRI Project No. 14936.12
xii

SwRI Project No. 14936.12

Table of Contents

Forward........................................................................................................................................... ii

Executive summary........................................................................................................................ iv

Table of Contents........................................................................................................................ xii

List of Figures............................................................................................................................... xv

List of Tables ............................................................................................................................... xxi

1 Introduction......................................................................................................................... 1

2 Modeling Approach ............................................................................................................ 2

2.1 Purpose of Model.................................................................................................................. 2

2.2 Model Improvement.............................................................................................................. 3

2.3 Monte Carlo Simulation Approach....................................................................................... 4

2.4 Calculation Methods............................................................................................................. 4

2.4.1 Calculation Method 1 – “Exhaust Flow-Torque-Speed” Method.................................... 5

2.4.2 Calculation Method 2 – “Exhaust and Fuel Flow-Torque-Speed” Method ..................... 6

2.4.3 Calculation Method 3 – “Fuel Flow-Torque-Speed” Method.......................................... 7

2.5 Reference NTE Events.......................................................................................................... 8

2.6 Error Surface Generation .................................................................................................... 14

2.7 Error Surface Sampling and Interpolation .......................................................................... 22

2.8 Brake-Specific Emissions Calculations.............................................................................. 25

2.9 Convergence and Number of Trials.................................................................................... 28

2.10 Simulation Output .......................................................................................................... 29

2.11 Step-by-Step Simulation Example ................................................................................. 29

2.12 Measurement Allowance Generation............................................................................. 33

2.12.1 Regression Method......................................................................................................... 33

2.12.2 Median Method .............................................................................................................. 34

2.13 Model Validation............................................................................................................ 35

3 Part 1065 PEMS and Laboratory Audit............................................................................ 39

3.1 1065 PEMS Audit............................................................................................................... 39

3.1.1 Horiba Flow Audits........................................................................................................ 40

3.1.2 Exhaust Flow.................................................................................................................. 45

4 Engine Dynamometer Testing and Results....................................................................... 47

4.1 Testing Objective ................................................................................................................ 47

4.2 Experimental Setup............................................................................................................. 48

4.2.1 Engine and Sampling System......................................................................................... 48

4.2.2 Sensors PPMD ............................................................................................................... 52

4.2.3 AVL MSS....................................................................................................................... 53

4.2.4 Horiba TRPM................................................................................................................. 55

4.3 Bypass Mixing Verification................................................................................................ 56

4.4 PEMS Loss Corrections...................................................................................................... 59

4.4.1 Sensors PPMD Loss Correction..................................................................................... 59

4.4.2 AVL MSS Loss Correction............................................................................................ 59

4.4.3 AVL MSS Total PM Correction .................................................................................... 60

4.5 Steady State Testing Procedure .......................................................................................... 60

4.6 Data Yield During Steady State Testing............................................................................. 69

4.6.1 Data Yield During Steady State Testing ........................................................................ 70

4.7 Accounting for CVS Variability During Steady State Testing........................................... 73
xiii

SwRI Project No. 14936.12

4.8 Steady State Testing Results............................................................................................... 75

4.9 Steady State PM Error Surfaces.......................................................................................... 87

4.10 Transient Engine Results................................................................................................ 92

4.11 CE-CERT Mobile Lab Correlation .............................................................................. 104

4.12 Investigation of DPF Regeneration.............................................................................. 107

4.13 Investigation of Storage and Release ........................................................................... 109

4.14 Engine Manufacturers Torque and Fuel Error Surfaces............................................... 112

5 Environmental Testing and Results................................................................................ 114

5.1 Reference Measurement Testing....................................................................................... 114

5.2 Pressure Chamber Testing ................................................................................................ 120

5.3 Temperature & Humidity Chamber Testing..................................................................... 129

5.4 Electromagnetic and Radio Frequency Interference Screening........................................ 138

5.4.1 Bulk Current Injection.................................................................................................. 140

5.4.2 Radiated Immunity....................................................................................................... 147

5.4.3 Electrostatic Discharge................................................................................................. 150

5.4.4 Conducted Transients................................................................................................... 151

5.5 Vibration Testing .............................................................................................................. 153

6 Modeling results.............................................................................................................. 162

6.1 Convergence Results from MC Runs ............................................................................... 163

6.2 Sensitivity Based on Bias and Variance ........................................................................... 168

6.3 Validation Results............................................................................................................. 183

6.4 Measurement Error Allowance Results............................................................................ 194

7 Summary......................................................................................................................... 215

7.1 Monte Carlo Model..............................................................Error! Bookmark not defined.

7.2 Engine Experiments and Results .........................................Error! Bookmark not defined.

7.2.1 Steady-State.....................................................................Error! Bookmark not defined.

7.2.2 Transient..........................................................................Error! Bookmark not defined.

7.2.3 CE-CERT and SwRI Correlation ....................................Error! Bookmark not defined.

8 References....................................................................................................................... 215

Appendix A: PEMS Operation Log............................................................................................ 217

Appendix B: crystal ball output file descriptions ....................................................................... 229

Appendix C: Monte Carlo Spreadsheet Computations............................................................... 241

Appendix D: Steering Committee Meeting Minutes.................................................................. 263
xiv

SwRI Project No. 14936.12
xv

SwRI Project No. 14936.12

LIST OF FIGURES

FIGURE 1. IN-USE VALIDATION PLOT FOR SENSORS PPMD USING METHOD 1. (Y AXIS IS THE DIFFERENCE BETWEEN 

PEMS AND IDEAL BSPM. IDEAL BSPM IS THE LABORATORY CVS BSPM FOR THE 5TH AND 95TH AND THE CECERT BSPM FOR THE DOTS) .............................................................................................................................. VII

FIGURE 2. IN-USE VALIDATION PLOT FOR AVL MSS USING METHOD 3. (Y AXIS IS THE DIFFERENCE BETWEEN PEMS 

AND IDEAL BSPM. IDEAL BSPM IS THE LABORATORY CVS BSPM FOR THE 5TH AND 95TH AND THE CE-CERT 

BSPM FOR THE DOTS) ...................................................................................................................................... VIII

FIGURE 3. METHOD 1 IDEAL BSPM VALUES FOR REFERENCE NTE EVENTS ................................................................11

FIGURE 4. ERROR SURFACE CONSTRUCTION: PEMS VS. LABORATORY RESULTS ......................................................17

FIGURE 5. ERROR SURFACE CONSTRUCTION: (PEMS - LAB) VS. LABORATORY RESULTS...........................................18

FIGURE 6. ERROR SURFACE CONSTRUCTION: ERROR AT VARIABILITY INDEX FOR 5TH AND 95TH PERCENTILES VS. 

LABORATORY RESULTS.......................................................................................................................................19

FIGURE 7. TRUNCATED NORMAL DISTRIBUTION PERCENTILES..................................................................................20

FIGURE 8. ERROR SURFACE CONSTRUCTION: ERROR AT VARIABILITY INDEX FOR 1ST AND 99TH PERCENTILES VS. 

LABORATORY RESULTS.......................................................................................................................................22

FIGURE 9. TRUNCATED STANDARD NORMAL AT 1ST AND 99TH PERCENTILES AND UNIFORM PROBABILITY DENSITY 

FUNCTIONS ........................................................................................................................................................23

FIGURE 10. STEADY-STATE PM ERROR SURFACE FOR AVL WITH EXAMPLE SAMPLING FOR A REFERENCE NTE EVENT

...........................................................................................................................................................................25

FIGURE 11. OVERVIEW OF MONTE CARLO SIMULATION FOR BSPM..........................................................................30

FIGURE 12. ERROR SURFACES INCLUDED IN MONTE CARLO SIMULATION................................................................32

FIGURE 13. LINEAR REGRESSION FIT TO 5TH AND 95TH PERCENTILE DELTAS ...........................................................38

FIGURE 14. LOESS REGRESSION FIT TO 5TH AND 95TH PERCENTILE DELTAS.............................................................39

FIGURE 15. LINEARITY CHECK ON PEMS-1 EXHAUST FLOW DURING STEADY STATE ENGINE TESTING......................45

FIGURE 16. LINEARITY CHECK ON PEMS-2 EXHAUST FLOW DURING STEADY STATE ENGINE TESTING......................46

FIGURE 17. LINEARITY CHECK ON PEMS-3 EXHAUST FLOW DURING STEADY STATE ENGINE TESTING......................47

FIGURE 18. VOLVO MP7 INSTALLED IN A CVS TEST CELL.............................................................................................48

FIGURE 19. DPF BYPASS WITH DOC.............................................................................................................................49

FIGURE 20. SCHEMATIC OF ENGINE DYNAMOMETER EXPERIMENTAL SETUP............................................................50

FIGURE 21. PPMD INSTALLED IN THE TEST CELL .........................................................................................................52

FIGURE 22. THE AVL MSS.............................................................................................................................................53

FIGURE 23. THE PROBE AND TPA FOR THE HORIBA OBS-TRPM..................................................................................54

FIGURE 24. THE HORIBA OBS-TRPM AND OBS-2200 ...................................................................................................56

FIGURE 25. EXPERIMENTAL SETUP FOR MIXING VERIFICATION..................................................................................57

FIGURE 26. MIXING VERIFICATION SAMPLE LOCATIONS ............................................................................................57

FIGURE 27. MIXING VERIFICATION RESULTS ...............................................................................................................58

FIGURE 28. SPEED AND LOAD FOR THE 80 POINTS CYCLE...........................................................................................61

FIGURE 29. FINAL SIX STEADY STATE MODES..............................................................................................................63

FIGURE 30. EXAMPLE OF STEADY STATE CYCLE...........................................................................................................65

FIGURE 31. STEADY STATE SAMPLE TIMES..................................................................................................................66

FIGURE 32. CVS FILTER WEIGHT GAIN FOR STEADY STATE TESTING...........................................................................67

FIGURE 33. STEADY STATE EXHAUST PM CONCENTRATION (μG/MOL) ......................................................................68

FIGURE 34. STEADY STATE BRAKE-SPECIFIC PM, CVS FILTER (MG/HP-HR) .................................................................69

FIGURE 35. NUMBER OF VALID DATA POINTS FOR STEADY STATE TESTING...............................................................70

FIGURE 36. HORIBA SAMPLE FLOW TO THE FILTER AND DILUTION FLOW WHILE COMPRESSOR STOPS ...................72
xvi

SwRI Project No. 14936.12

FIGURE 37. SENSORS EFM ZERO DURING STEADY STATE CYCLE .................................................................................73

FIGURE 38. HORIBA-1 STEADY STATE PM CONCENTRATION DELTAS .........................................................................76

FIGURE 39. SENSORS-1 STEADY STATE PM CONCENTRATION DELTAS .......................................................................76

FIGURE 40. AVL PM CONCENTRATION DELTAS FOR STEADY STATE TESTING ON PEMS 1 ..........................................77

FIGURE 41. HORIBA-2 STEADY STATE PM CONCENTRATION DELTAS .........................................................................78

FIGURE 42. SENSORS-2 STEADY STATE PM CONCENTRATION DELTAS .......................................................................78

FIGURE 43. AVL-2 STEADY STATE PM CONCENTRATION DELTAS ................................................................................79

FIGURE 44. HORIBA-3 STEADY STATE PM CONCENTRATION DELTAS .........................................................................80

FIGURE 45. SENSORS-3 STEADY STATE PM CONCENTRATION DELTAS .......................................................................80

FIGURE 46. AVL-3 STEADY STATE PM CONCENTRATION DELTAS ................................................................................81

FIGURE 47. STEADY STATE CONCENTRATION DELTAS FOR HORIBA............................................................................82

FIGURE 48. STEADY STATE CONCENTRATION DELTAS FOR SENSORS .........................................................................82

FIGURE 49. STEADY STATE CONCENTRATION DELTAS FOR AVL ..................................................................................83

FIGURE 50. STEADY STATE HORIBA PEMS PM CONCENTRATION VERSUS THE LABORATORY REFERENCE.................84

FIGURE 51. STEADY STATE SENSORS PEMS PM CONCENTRATION VERSUS THE LABORATORY REFERENCE...............85

FIGURE 52. STEADY STATE AVL PEMS PM CONCENTRATION VERSUS THE LABORATORY REFERENCE .......................86

FIGURE 53. LINEAR REGRESSION CORRELATION BETWEEN PEMS AND LAB...............................................................87

FIGURE 54. STEADY STATE CONCENTRATION DELTAS FOR HORIBA............................................................................88

FIGURE 55. STEADY STATE CONCENTRATION DELTAS FOR SENSORS .........................................................................88

FIGURE 56. STEADY STATE CONCENTRATION DELTAS FOR AVL ..................................................................................89

FIGURE 57. FINAL STEADY STATE PM ERROR SURFACE – HORIBA ..............................................................................90

FIGURE 58. FINAL STEADY STATE PM ERROR SURFACE - SENSORS .............................................................................91

FIGURE 59. FINAL STEADY STATE PM ERROR SURFACE – AVL .....................................................................................91

FIGURE 60. REPEAT ENGINE SPEED TRACES FOR NTE TRANSIENT CYCLE....................................................................93

FIGURE 61. REPEAT ENGINE TORQUE TRACES FOR NTE TRANSIENT CYCLE................................................................94

FIGURE 62. REPEAT AVL SOOT CONCENTRATION TRACES FOR NTE TRANSIENT CYCLE..............................................95

FIGURE 63. AVL SOOT CONCENTRATION DURING NTE TRANSIENT CYCLES, EVENTS 20-23 .......................................96

FIGURE 64. COMPARISON OF AVL AND HORIBA REAL TIME SIGNALS DURING TRANSIENT CYCLE.............................97

FIGURE 65. CVS BSPM AND CORRECTION FACTOR FOR TRANSIENT CYCLE ................................................................98

FIGURE 66. HORIBA CONCENTRATION DELTAS FOR TRANSIENT ENGINE TESTING....................................................99

FIGURE 67. SENSORS CONCENTRATION DELTAS FOR TRANSIENT ENGINE TESTING.................................................100

FIGURE 68. AVL CONCENTRATION DELTAS FOR TRANSIENT ENGINE TESTING.........................................................101

FIGURE 69. FINAL HORIBA TRANSIENT PM ERROR SURFACE ....................................................................................102

FIGURE 70. FINAL SENSORS TRANSIENT PM ERROR SURFACE ..................................................................................103

FIGURE 71. FINAL AVL TRANSIENT PM ERROR SURFACE...........................................................................................103

FIGURE 72. EXHAUST CONFIGURATION FOR CE-CERT CORRELATION.......................................................................104

FIGURE 73. BRAKE-SPECIFIC PM RESULTS FROM CE-CERT CORRELATION ................................................................106

FIGURE 74. CVS FILTER WEIGHT GAIN DURING TUNNEL BLANKS .............................................................................107

FIGURE 75. BRAKE SPECIFIC PM EMISSIONS DURING ACTIVE REGENERATION.........................................................108

FIGURE 76. TOTAL EXHAUST NUMBER CONCENTRATION DURING STORAGE AND RELEASE....................................110

FIGURE 77. BRAKE-SPECIFIC PM EMISSIONS DURING STORAGE AND RELEASE INVESTIGATION..............................111

FIGURE 78. OEM SUPPLIED TORQUE ERRORS ...........................................................................................................112

FIGURE 79. OEM SUPPLIED FUEL FLOW ERRORS.......................................................................................................113

FIGURE 80. TARGET DILUTION RATIO AND PM LEVEL PROFILE FOR ENVIRONMENTAL TESTING.............................115

FIGURE 81. EXPERIMENTAL SETUP FOR ENVIRONMENTAL TESTING ........................................................................116

FIGURE 82. HORIBA ENVIRONMENTAL BASELINE MEASUREMENTS.........................................................................117
xvii

SwRI Project No. 14936.12

FIGURE 83. SENSORS ENVIRONMENTAL BASELINE MEASUREMENTS.......................................................................118

FIGURE 84. AVL ENVIRONMENTAL BASELINE MEASUREMENTS ...............................................................................118

FIGURE 85. REFERENCE MSS ENVIRONMENTAL BASELINE MEASUREMENTS ...........................................................119

FIGURE 86. REFERENCE AVL VERSUS PEMS AVL FOR ENVIRONMENTAL BASELINE ..................................................120

FIGURE 87. ORIGINAL AND REVISED PROFILE FOR ALTITUDE TESTING.....................................................................121

FIGURE 88. ALTITUDE TESTING CHAMBER ................................................................................................................122

FIGURE 89. PEMS INSTALLED IN THE ALTITUDE CHAMBER.......................................................................................123

FIGURE 90. HORIBA ENVIRONMENTAL PRESSURE MEASUREMENTS........................................................................124

FIGURE 91. SENSORS ENVIRONMENTAL PRESSURE MEASUREMENTS......................................................................125

FIGURE 92. AVL ENVIRONMENTAL PRESSURE MEASUREMENTS ..............................................................................126

FIGURE 93. AVL PRESSURE MEDIAN VERSUS MAD....................................................................................................127

FIGURE 94. HORIBA PRESSURE MEDIAN VERSUS MAD .............................................................................................127

FIGURE 95. SENSORS PRESSURE MEDIAN VERSUS MAD ...........................................................................................128

FIGURE 96. FINAL ERROR SURFACE FOR ENVIRONMENTAL PRESSURE AVL PM CONCENTRATION..........................129

FIGURE 97. TEMPERATURE AND HUMIDITY PROFILE FOR ENVIRONMENTAL TESTING............................................130

FIGURE 98. PEMS INSTALLED IN THE TEMPERATURE AND HUMIDITY CHAMBER.....................................................131

FIGURE 99. THE PM GENERATOR INSTALLED OUTSIDE THE TEMPERATURE AND HUMIDITY CHAMBER .................132

FIGURE 100. HORIBA ENVIRONMENTAL TEMPERATURE MEASUREMENTS..............................................................133

FIGURE 101. SENSORS ENVIRONMENTAL TEMPERATURE MEASUREMENTS............................................................134

FIGURE 102. AVL ENVIRONMENTAL TEMPERATURE MEASUREMENTS ....................................................................135

FIGURE 103. HORIBA TEMPERATURE MEDIAN VERSUS MAD ...................................................................................136

FIGURE 104. SENSORS TEMPERATURE MEDIAN VERSUS MAD .................................................................................136

FIGURE 105. HORIBA TEMPERATURE AND HUMIDITY MEDIAN VERSUS MAD .........................................................137

FIGURE 106. FINAL ERROR SURFACE ENVIRONMENTAL TEMPERATURE AND HUMIDITY AVL PM...........................138

FIGURE 107. AVL PEMS IN THE RADIATION CHAMBER FOR EMI AND RFI TESTING ..................................................140

FIGURE 108. BULK CURRENT INJECTION PROBE........................................................................................................141

FIGURE 109. HORIBA PEMS SETUP FOR BULK CURRENT INJECTION.........................................................................142

FIGURE 110. SENSORS SETUP DURING BULK CURRENT INJECTION TESTING............................................................143

FIGURE 111. AVL SETUP DURING BULK CURRENT INJECTION TESTING.....................................................................143

FIGURE 112. HORIBA FLOW DISTURBANCE FROM BULK CURRENT INJECTION.........................................................144

FIGURE 113. HORIBA EXHAUST FLOW NOISE ON ANALOG CABLE DURING BULK CURRENT INJECTION ..................145

FIGURE 114. BCI NOISE ON AVL ANALOG OUTPUT CABLE ........................................................................................147

FIGURE 115. HORIBA PEMS SETUP DURING RADIATED IMMUNITY TESTING ...........................................................148

FIGURE 116. SENSORS PEMS SETUP DURING RADIATED IMMUNITY TESTING .........................................................149

FIGURE 117. AVL PEMS SETUP DURING RADIATED IMMUNITY TESTING..................................................................149

FIGURE 118. HORIBA DILUTION RATIO FLUCTUATIONS DURING RADIATED IMMUNITY..........................................150

FIGURE 119. ELECTROSTATIC DISCHARGE SIMULATOR.............................................................................................151

FIGURE 120. SENSORS SETUP DURING CONDUCTED TRANSIENTS TESTING.............................................................152

FIGURE 121. AVL SETUP DURING CONDUCTED TRANSIENTS TESTING .....................................................................152

FIGURE 122. SENSORS PEMS MOUNTED FOR TRANSVERSE HORIZONTAL VIBRATION.............................................154

FIGURE 123. SENSORS PEMS MOUNTED FOR LONGITUDINAL HORIZONTAL VIBRATION.........................................154

FIGURE 124. SENSORS PEMS MOUNTED FOR TRANSVERSE 45° VIBRATION.............................................................155

FIGURE 125. SENSORS PEMS MOUNTED FOR LONGITUDINAL 45° VIBRATION.........................................................155

FIGURE 126. SENSORS PEMS MOUNTED FOR VERTICAL VIBRATION ........................................................................156

FIGURE 127. SENSORS PEMS MOUNTED FOR 45° VERTICAL VIBRATION..................................................................156

FIGURE 128. POWER SPECTRAL DENSITY FOR VERTICAL VIBRATION TESTING .........................................................157
xviii

SwRI Project No. 14936.12

FIGURE 129. POWER SPECTRAL DENSITY FOR HORIZONTAL VIBRATION TESTING ...................................................158

FIGURE 130. HORIBA PEMS VIBRATION POSITIONS ..................................................................................................159

FIGURE 131. SENSORS TOTAL FLOW DURING VIBRATION TESTING..........................................................................160

FIGURE 132. AVL PEMS VIBRATION POSITIONS.........................................................................................................161

FIGURE 133. AVL SOOT MEASUREMENT NOISE DURING VIBRATION TESTING.........................................................162

FIGURE 134. CONVERGENCE FOR AVL METHOD 1 AS A PERCENT OF BSPM THRESHOLD .......................................164

FIGURE 135. CONVERGENCE FOR AVL METHOD 2 AS A PERCENT OF BSPM THRESHOLD .......................................165

FIGURE 136. CONVERGENCE FOR AVL METHOD 3 AS A PERCENT OF BSPM THRESHOLD .......................................165

FIGURE 137. CONVERGENCE FOR HORIBA METHOD 1 AS A PERCENT OF BSPM THRESHOLD.................................166

FIGURE 138. CONVERGENCE FOR HORIBA METHOD 2 AS A PERCENT OF BSPM THRESHOLD.................................167

FIGURE 139. CONVERGENCE FOR SENSORS METHOD 1 AS A PERCENT OF BSPM THRESHOLD...............................167

FIGURE 140. CONVERGENCE FOR SENSORS METHOD 2 AS A PERCENT OF BSPM THRESHOLD...............................168

FIGURE 141. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND VARIANCE FOR AVL BSPM METHOD 

1 .......................................................................................................................................................................174

FIGURE 142. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND VARIANCE FOR AVL METHOD 2....175

FIGURE 143. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND VARIANCE FOR AVL METHOD 3....176

FIGURE 144. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND VARIANCE FOR HORIBA BSPM 

METHOD 1........................................................................................................................................................177

FIGURE 145. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND VARIANCE FOR HORIBA BSPM 

METHOD 2........................................................................................................................................................178

FIGURE 146. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND VARIANCE FOR SENSORS BSPM 

METHOD 1........................................................................................................................................................179

FIGURE 147. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND VARIANCE FOR SENSORS BSPM 

METHOD 2........................................................................................................................................................180

FIGURE 148. VALIDATION PERCENTILES FOR THE 141 REFERENCE NTE EVENTS FOR AVL METHOD 1 ....................184

FIGURE 149. VALIDATION PERCENTILES FOR THE 141 REFERENCE NTE EVENTS FOR AVL METHOD 2 ....................185

FIGURE 150. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS FOR AVL METHOD 3 ...........................185

FIGURE 151. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS FOR HORIBA METHOD 1.....................186

FIGURE 152. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS FOR HORIBA METHOD 2.....................186

FIGURE 153. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS FOR SENSORS METHOD 1...................187

FIGURE 154. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS FOR SENSORS METHOD 2...................187

FIGURE 155. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR SENSORS METHOD 1 ..........................189

FIGURE 156. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR SENSORS METHOD 1 ............................190

FIGURE 157. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR SENSORS METHOD 2 ..........................190

FIGURE 158. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR SENSORS METHOD 2 ............................191

FIGURE 159. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR AVL METHOD 1...................................191

FIGURE 160. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR AVL METHOD 1.....................................192

FIGURE 161. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR AVL METHOD 2...................................192

FIGURE 162. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR AVL METHOD 2.....................................193

FIGURE 163. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR AVL METHOD 3...................................193

FIGURE 164. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR AVL METHOD 3.....................................194

FIGURE 165. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS IDEAL BSPM FOR AVL METHOD 1 .....195

FIGURE 166. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS IDEAL BSPM FOR AVL METHOD 2 .....197

FIGURE 167. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS IDEAL BSPM FOR AVL METHOD 3 .....198

FIGURE 168. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS IDEAL BSPM FOR HORIBA METHOD 1

.........................................................................................................................................................................200
xix

SwRI Project No. 14936.12

FIGURE 169. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS IDEAL BSPM FOR HORIBA METHOD 2

.........................................................................................................................................................................201

FIGURE 170. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS IDEAL BSPM FOR SENSORS METHOD 1

.........................................................................................................................................................................203

FIGURE 171. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS IDEAL BSPM FOR SENSORS METHOD 2

.........................................................................................................................................................................204

FIGURE 172. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

SENSORS METHOD 1 WITH NO REGEN ............................................................................................................207

FIGURE 173. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

SENSORS METHOD 2 WITH NO REGEN ............................................................................................................208

FIGURE 174. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

AVL METHOD 1 WITH NO REGEN.....................................................................................................................209

FIGURE 175. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

AVL METHOD 2 WITH NO REGEN.....................................................................................................................210

FIGURE 176. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

AVL METHOD 3 WITH NO REGEN.....................................................................................................................211

FIGURE 177. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

AVL METHOD 1 WITH REGEN...........................................................................................................................212

FIGURE 178. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

AVL METHOD 2 WITH REGEN...........................................................................................................................213

FIGURE 179. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON THE SIMULATION MODEL FOR BSPM 

AVL METHOD 3 WITH REGEN...........................................................................................................................214

FIGURE 180. SENSITIVITY CHART FOR AVL DELTA PM METHOD 1 ...........................................................................240

FIGURE 181. BRAKE-SPECIFIC PM BY METHOD 1......................................................................................................257
xx

SwRI Project No. 14936.12
xxi

SwRI Project No. 14936.12

LIST OF TABLES

TABLE 1. MEASRUEMENT ALLOWANCE BASED ON 0.02 G/HP-HR THRESHOLD ......................................................... VI

TABLE 2. SENSORS PPMD VALIDATION RESULTS..........................................................................................................IX

TABLE 3. AVL MSS VALIDATION RESULTS .....................................................................................................................IX

TABLE 4. PM-PEMS USED ALONG WITH SERIAL NUMBER.............................................................................................2

TABLE 5. REFERENCE NTE EVENTS AND IDEAL BSPM EMISSIONS ................................................................................9

TABLE 6. DESCRIPTIVE STATISTICS FOR BSPM EMISSIONS FOR 141 REFERENCE NTE EVENTS...................................11

TABLE 7. INPUT PARAMETERS FOR REFERENCE NTE EVENTS.....................................................................................13

TABLE 8. ERROR SURFACES FOR MONTE CARLO SIMULATION ..................................................................................15

TABLE 9. ERROR SURFACES USED FOR COMPUTING BRAKE-SPECIFIC PM EMISSIONS BY THREE CALCULATION 

METHODS...........................................................................................................................................................27

TABLE 10. EXAMPLE OF SELECTION OF MEASUREMENT ALLOWANCE AT 0.02 G/HP-HR NTE THRESHOLD FOR THE 

AVL PEMS ...........................................................................................................................................................35

TABLE 11. SUMMARY OF PART 1065 AUDITS ..............................................................................................................40

TABLE 12. LINEARITY VERIFICATIONS FOR HORIBA PEMS ...........................................................................................41

TABLE 13. LINEARITY VERIFICATIONS FOR SENSORS PEMS .........................................................................................43

TABLE 14. LINEARITY VERIFIC ATIONS FOR AVL PEMS.................................................................................................44

TABLE 15. LIST OF DPF BYPASS CONFIGURATIONS......................................................................................................50

TABLE 16. SAMPLE ORDER FOR STEADY STATE CYCLE TESTING..................................................................................64

TABLE 17. DATA YIELD BY EACH PM-PEMS..................................................................................................................71

TABLE 18. TEST PROCEDURE FOR CE-CERT CORRELATION........................................................................................105

TABLE 19. PM EMISSIONS RESULTS FROM ACTIVE DPF REGENERATION..................................................................108

TABLE 20. AVERAGE BRAKE-SPECIFIC EMISSIONS DURING STORAGE AND RELEASE CYCLE......................................111

TABLE 21. OEM ERROR SURFACE DELTAS FOR TORQUE AND FUEL FLOW................................................................113

TABLE 22. SUMMARY OF NUMBER OF REFERENCE NTES MEETING 2% CONVERGENCE .........................................168

TABLE 23. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 REFERENCE NTE EVENTS FOR AVL BSPM 

METHOD 1........................................................................................................................................................170

TABLE 24. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 REFERENCE NTE EVENTS FOR AVL BSPM

METHOD 2 .......................................................................................................................................................170

TABLE 25. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 REFERENCE NTE EVENTS FOR AVL BSPM 

METHOD 3........................................................................................................................................................171

TABLE 26. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 REFERENCE NTE EVENTS FOR HORIBA 

BSPM METHOD 1 .............................................................................................................................................171

TABLE 27. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 REFERENCE NTE EVENTS FOR HORIBA 

BSPM METHOD 2 .............................................................................................................................................172

TABLE 28. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 REFERENCE NTE EVENTS FOR SENSORS 

BSPM METHOD 1 .............................................................................................................................................172

TABLE 29. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 REFERENCE NTE EVENTS FOR SENSORS 

BSPM METHOD 2 .............................................................................................................................................173

TABLE 30. SUMMARY OF ERROR SURFACE SENSITIVITIES TO BIAS AND VARIANCE FOR BSPM METHOD 1 ............181

TABLE 31. SUMMARY OF ERROR SURFACE SENSITIVE TO BIAS AND VARIANCE FOR BSPM METHOD 2..................182

TABLE 32. SUMMARY OF ERROR SURFACE SENSITIVE TO BIAS AND VARIANCE FOR BSPM METHOD 3..................183

TABLE 33. LOESS SMOOTHING PARAMETERS FOR VALIDATION PERCENTILES ........................................................188

TABLE 34. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING REGRESSION AND MEDIAN METHODS FOR 

AVL METHOD 1.................................................................................................................................................196
xxii

SwRI Project No. 14936.12

TABLE 35. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING REGRESSION AND MEDIAN METHODS FOR 

AVL METHOD 2.................................................................................................................................................197

TABLE 36. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING REGRESSION AND MEDIAN METHODS FOR 

AVL METHOD 3.................................................................................................................................................199

TABLE 37. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING REGRESSION AND MEDIAN METHODS FOR 

HORIBA METHOD 1 ..........................................................................................................................................200

TABLE 38. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING REGRESSION AND MEDIAN METHODS FOR 

HORIBA METHOD 2 ..........................................................................................................................................202

TABLE 39. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING REGRESSION AND MEDIAN METHODS FOR 

SENSORS METHOD 1 ........................................................................................................................................203

TABLE 40. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING REGRESSION AND MEDIAN METHODS FOR 

SENSORS METHOD 2 ........................................................................................................................................205

TABLE 41. BSPM MEASUREMENT ERROR IN PERCENT OF NTE THRESHOLD BY PEMS AND CALCULATION METHOD

.........................................................................................................................................................................205

TABLE 42. MEASUREMENT ALLOWANCE AT NTE THRESHOLD BY EMISSIONS FOR METHOD 2...............................206

TABLE 43. SUMMARY OF BSPM MODEL VALIDATION RESULTS ...............................................................................214

TABLE 44. SIMULATION VARIABLES..........................................................................................................................230

TABLE 45. DESCRIPTIVE STATISTICS FOR SIMULATION VARIABLES ..........................................................................236

TABLE 46. ERROR SURFACES USED IN SIMULATION..................................................................................................246

TABLE 47. METHODS WORKSHEET PARAMETER COLUMN DESCRIPTIONS ..............................................................250
1

SwRI Project No. 14936.12

1 INTRODUCTION

The U.S. EPA, EMA, and CARB agreed to pursue an experimental data driven program to 

determine PEMS bias and precision measurement errors expected in in-use NTE testing and 

compliance before enforcement. The idea is to combine these errors into a measurement 

allowance and add it to the EPA in-use regulatory limit to increase it to a new threshold level that 

the engine manufacturers must comply with. The gaseous PEMS measurement allowance 

program was completed by SwRI in April of 2007, and this work focuses on the PM-PEMS

measurement allowance.

To determine all bias and precision errors associated with in-use PM-PEMS measurement, it will 

take a very extensive set of experimental data and engines to cover all engine steady-state and 

transient NTE operations as well as different environmental conditions and configurations that 

may influence the measurement. Instead of focusing on a wide matrix of experimental data, the 

SC approach was to:

a) Perform repeats on a series of laboratory on a six NTE steady-state tests using the CVS 

PM filter method and the PM-PEMS, and establish a bias error surface for each the 

PEMS tested

b) Perform repeats on a series of laboratory NTE transient cycles containing thirty 32 

seconds NTE events, and use the PM-PEMS measurement to produce precision error 

surface for each of the PEMS tested

c) Perform a series of environmental tests that includes electromagnetic and radio 

frequency interferences, shock and vibration, pressure, and temperature and humidity, 

and produce an error surface, if any, for each PEMS associated with each parameter

d) Use torque and fuel flow error surfaces provided by the engine manufacturers

e) Use other error surfaces established during the gaseous measurement allowance program

f) Use Monte-Carlo simulation based on a set of 141 Ideal brake-specific PM (BSPM) 

reference NTE events to predict the error distribution at each reference NTE

g) Use the 95th percentile and 5th

h) Perform actual in-use testing with the PM-PEMS using the CE-CERT trailer to 

determine whether or not the data generated in-use validates the model.

percentile of the error distribution at each Ideal BSPM 

reference NTE level as the upper and low boundary of the deltas between PM-PEMS 

BSPM and Ideal BSPM.

A total of nine PM-PEMS, three from each manufacturer, were used on this program. The PMPEMS, shown in Table 4, included the Sensors PPMD, Horiba TRPM, and AVL MSS. Each set 

of three PM-PEMS, one PM-PEMS from each manufacturer, was tested simultaneously in 

parallel using a series of steady-state and transient NTE engine experiments. After the 

completion of steady-state testing and transient testing, one PM-PEMS from each manufacturer 

was used on a series of environmental test conditions that included electromagnetic and radio 

frequency interferences; shock and vibration, atmospheric pressure, and temperature and 

humidity.
2

SwRI Project No. 14936.12

TABLE 4. PM-PEMS USED ALONG WITH SERIAL NUMBER

PEMS Name PEMS Serial Number

Horiba 1 10107-01

Horiba 2 10107-02

Horiba 3 10107-03

Sensors 1 E08-PD03

Sensors 2 G08-PD02

Sensors 3 A08-PD03

AVL 1 346

AVL 2 472

AVL 3 273

To validate the model, a series of in-use tests with the PM-PEMS were performed by CE-CERT 

using the CE-CERT emissions trailer.

This report describes: 

• Model approach used to perform the modeling portion of this work. 

• Engine experiments with the PM-PEMS including steady-state and transient NTE PM 

results

• Environmental setup and PM results

• Measurement allowance produced by the model for each of the three PM-PEMS provided 

by each manufacturer

• Model validation using the PM-PEMS selected for in-use testing 

2 MODELING APPROACH

2.1 PURPOSE OF MODEL

The desire for this program was to generate BSPM measurement allowances based on rigorous 

statistical methods applied to a large body of data. At the same time, it was desirable to exclude 

outlier data caused by extreme measurement errors which were not considered representative of 

normal in-use operations. A direct approach could have been to test PEMS against some kind of 

mobile laboratory reference (such as the CE-CERT Mobile Emission Laboratory) on a large 

number of vehicles, and quantify errors directly. However, such an approach would have been 

expensive in terms of both time and funding.

Given these factors, the Steering Committee ultimately elected to use a simulation approach in 

order to generate the BSPM measurement allowances, similar to what was done in the gaseous 

in-use emissions testing program [1]. In this approach, the Steering Committee defined all of the 

expected sources of PEMS measurement errors based on existing in-use testing expertise and 
3

SwRI Project No. 14936.12

understanding of how the PEMS functioned. Each of these errors was quantified using a series 

of controlled laboratory experiments, each designed to isolate errors related to a single error 

source. The results of each experiment would essentially be an empirical model of a given 

source of measurement error. In this report, these error models are referred to as error surfaces. 

It is important to note that each of these error surfaces represents an incremental error of PEMS 

measurement, as compared to an associated laboratory reference measurement.

2.2 MODEL IMPROVEMENT

Several improvements to the execution of the Monte Carlo simulation model were implemented 

to improve the efficiency and post-processing of the simulation runs. Eight macros were written 

to perform various tasks and are summarized in the section below.

Macro 1: Controls batch processing and allows the ability to run several simulations backto-back (batch mode). Reads each reference NTE event data, number of trials, and the number 

of reference NTE events in the batch run. Calls other macros.

Macro 2: Clears and deletes extra rows in error model (see Appendix D for a detailed 

description). Calculates ideal PM emissions for each calculation method.

Macro 3: Checks error surfaces turned ‘off’ in simulation run and clears unused error 

surface charts and Excel tabs for calculation speedup (see Appendix D for a detailed 

description).

Macro 4: Controls Crystal Ball run preferences, runs Crystal Ball simulation for the given 

NTE Event, and controls Crystal Ball creation of Report and Extract data files. No longer stores

each trial ic value (40,000 – 65,000 values). Reduced the number of sensitivity charts created.

Only stores BS emissions in g/hp-hr units. This reduced EXTRACT and REPORT files from 

133 MB to 17 MB per NTE event (≈ 87% reduction). 

Macro 5: Reads EXTRACT file, controls calculation of 5th, 50th and 95th percentiles from 

full model and calls Macro 6.

Macro 6: Computes 5th, 50th and 95th percentile delta BS emissions and confidence limits 

from order statistics from the Full model. These values are used in the measurement allowance 

plots and to check for convergence.

Macro 7: Reads Extract file Validation data, computes 5th, 50th and 95th percentiles for the 

Validation model (calls Macro 6).

Macro 8: Reads REPORT file, selects and formats sensitivity data for Assumption 

sensitivities greater than 5 percent or less than -5 percent contribution to variance and stores in 

sensitivity file. Creates sensitivity file with “important” error surfaces.
4

SwRI Project No. 14936.12

The development of the macros resulted in substantial improvements in the simulation run-time 

and post-processing of the extract files. The estimated reduction in model simulation run-time 

and post processing was approximately 80 to 85 percent of the time used in the gaseous PEMS 

program. The developed macros eliminated the need to manually post-process the files as had 

been done in the gaseous PEMS program. The batch processing allowed up to 20 reference NTE 

events to be simulated in a single Excel run. The size of the EXTRACT and REPORT files was 

greatly reduced by only storing needed variables and delta PM emissions.

2.3 MONTE CARLO SIMULATION APPROACH

The error surfaces representing incremental errors of PEMS measurement were programmed into 

a computer model which employed Monte Carlo random sampling methods to simulate the 

combined effects of all of these sources of error on the final measured brake-specific value. An 

ideal reference NTE data set for a given test event was run through the Model, and all the various 

errors were applied to that data set in a randomly chosen manner. Brake-specific PM emission 

values were then calculated for both the ideal and error-applied data sets, which were compared 

to yield a final measurement error. The process was repeated thousands of times, with many 

different ideal data sets, to generate a large, robust data set which was evaluated to determine a 

final set of combined measurement errors. These final errors, referred to in this report as deltas, 

were generated for the PM pollutant for each calculation method and three difference PEMS 

model units, for a final set of seven deltas; Methods 1-3 for the AVL PEMS unit, Methods 1-2 

for the Horiba PEMS unit and Methods 1-2 for the Sensors PEMS unit. A complete description 

of the Monte Carlo methodology and of the model is given in Section 2.3 of this report.

2.4 CALCULATION METHODS 

Calculations must be performed on the recorded data to determine brake-specific PM emission 

values in accordance with methods outlined in 40 CFR Part 1065 Subparts G and J. The 

symbolic notation given in the formulas shown later in this section is fully described in 40 CFR 

Part 1065 Subpart K [2].

40 CFR Part 1065 allows for the use of any of three different calculation methods in order to 

determine brake-specific emission values from in-use test data. The basic calculation of brakespecific emissions requires three main inputs as follows:

BBBB EEEEEEEEEEEEEEEEEE = MM

WW WWWW = CC CCCC CC CC xx FFFF FF

PP
5

SwRI Project No. 14936.12

The three calculation methods vary somewhat in the means used to determine either the Flow 

component or the Work component of this calculation. Each of the three methods is summarized 

below. Because each method relies on different inputs, it is possible that each method of 

calculation will react differently to various measurement errors. Therefore, measurement 

allowances must be examined independently for each method. However, according to the Test 

Plan methodology, only one of the three calculation methods would be selected to generate the 

final measurement allowances. The selection methodology is outlined later in this introduction 

under the Measurement Allowance Generation section.

2.4.1 Calculation Method 1 – “Exhaust Flow-Torque-Speed” Method

Calculation Method 1 is analogous to the method used by most dynamometer laboratories, and 

relies on direct input of both exhaust flow and torque. In the case of exhaust flow, this is the 

flow rate measured by the same form of exhaust flow meter. The Sensors and AVL PEMS relied 

on the Sensors exhaust flow meter (EFM) while the Horiba PEMS had a tail pipe adapter (TPA) 

it employed for exhaust flow measurement. Work is not measured directly, but is instead 

calculated using ECM broadcast engine speed and ECM broadcast engine torque. While engine 

speed is directly measured by the engine ECM, ECM broadcast torque is an estimate based on a 

variety of other parameters, therefore, torque cannot be directly verified during in-use testing. A 

simplified formula for this method is:

h oo 1 = ∑ gg

∑

The more complete formula used for Method 1 is as follows:

For all PM PEMS:

m PM

( ) ( ) ( ) ∑

∑

=

=





 



 ⋅ ∆





 



  ∆ 

 



 



 





⋅ = N

i

i i

N

i

PM i

PM Speed rpm T N m t

t

s

mol

n

mol

g m

e g kW hr

1

1

60*1000*3600

* * 2*3.14159*

* *

/



is a flow weighted particulate matter exhaust concentration in g/mol 

Where for AVL:
6

SwRI Project No. 14936.12

m PM

1

1

*

*

N

i i

i

PM N

i

i

g mol mPM n t

g mol s

m

mol mol

n t

s

=

=

        ∗ ∆            =         ∆      

∑

∑





is computed numerically as follows,

It should be noted that calculation Method 1 is directly dependent on the accuracy of both the 

exhaust flow meter and the torque estimation, as well as on the measurement of gaseous 

concentration. 

2.4.2 Calculation Method 2 – “Exhaust and Fuel Flow-Torque-Speed” Method

This calculation is designated solely for in-use testing, and is designed to minimize the effect of 

errors related to the accuracy of the exhaust flow measurement. The Method 2 calculation adjusts 

the exhaust flow measurement by a ratio of the CO2-based fuel flow to the ECM reported fuel 

flow. This means that although the flow meter must be linear, it does not necessarily have to be 

accurate. In addition, Method 2depends on the ECM broadcasted torque and speed, and on the 

ratio of fuel flow calculated from the carbon balance using gaseous measurement over the fuel 

flow broadcasted by the ECM. 2 depends on the ECM broadcasted torque and speed. A 

simplified version of this method can be expressed as:

h oo 2 = ∑ gg

∑ � CCCC2 ffff

EE ffff × WW WWWW�

The more complete formula for Method 22 using PM as an example, is:
7

SwRI Project No. 14936.12

For all PM PEMS:

m PM

( )

( ) ( ( ) ( ))

( ) ( )

1

6 2

2

* *

/

* *10 % % *10 *

*

* *2*3.14159

60*1000*3600

i

N

PM i

i

PM

ii i

C

fuel

fueli

i i

g mol m nt

mol s e g kW hr

mol n xTHC ppm xCO xCO t

M s

w g m

s

Speed rpm T N m

=

− −

      ∆       ⋅ =  

 

      ++ ∆    

         

⋅

   

∑ 



 1

N

i=

∑

is a flow weighted particulate matter exhaust concentration in g/mol 

Where for AVL:

m PM

1

1

*

*

N

i i

i

PM N

i

i

g mol mPM n t

g mol s

m

mol mol

n t

s

=

=

        ∗ ∆            =         ∆      

∑

∑





is computed numerically as follows,

It should be noted that, as mentioned earlier, Method 22 is not subject to accuracy errors for the 

exhaust flow measurement, although that measurement must still be linear for the method to 

function properly.

2.4.3 Calculation Method 3 – “Fuel Flow-Torque-Speed” Method

Method 3 does not use direct measurement of exhaust flow, but relies on a carbon balance and 

ECM broadcast fuel rate to determine mass. The work term for Method 3 is determined

identically to the work term for Method 1; using the ECM broadcast values for engine speed and 

torque to calculate work. Method 3 entirely circumvents the use of an exhaust flow meter, but 

for the HDIUT program, EPA must approve the use of Method 1 for a given test and 

manufacturer. A simplified version of Method 1 may be expressed as:
8

SwRI Project No. 14936.12

h oo 3 =

∑ �gg ×

EE ffff

CCCC2 ffff �

∑ WW WWWW

The more complete formula for Method 3 is:

For AVL Only:

( )

( ) ( ( ) ( ))

( ) ( )

6 2

1 2

1

/

* * *

*10 % % *10

* *2*3.14159*

60*1000*3600

i

PM

N ifue l

fuel

PM

C i i i

N

i i

i

e g kW hr

g m

g w s m t

mol M xTHC ppm xCO xCO

Speed rpm T N m t

− − =

=

⋅ =

              ∆   + +

 

  ⋅ ∆

   

∑

∑



Where:

( ( ))

( ) ( ( ) ( ))

( ) ( ( ) ( ))

6 2

1 2

6 2

1 2

/ *

* *

*10 % % *10

* *

*10 % % *10

i

i

N i ifue l

fuel

C i i i

PM

N ifue l

fuel

C i i i

g mPM g mol m w s t

M xTHC ppm xCO xCO

g m

mol g m

w s t

M xTHC ppm xCO xCO

− − =

− − =

          ∆

+ +

      =             ∆

+ +

 

∑

∑





It should be noted that Method 3 is not subject to exhaust flow measurement accuracy errors, but 

also that this method is wholly dependent on ECM broadcast values for both mass and work 

determination.

2.5 REFERENCE NTE EVENTS

The reference data set to which all the simulated errors were applied represented engine 

operations over a wide range of NTE events. This reference data set was generated from 

collections of real-world PEMS data sets. Parameters in the reference data set were scaled in 
9

SwRI Project No. 14936.12

order to exercise the model through a more appropriate range of parameters (i.e. concentrations, 

flows, ambient conditions, etc.). In this scaling process, care was taken to maintain the dynamic 

characteristics of the reference data set.

The Monte Carlo simulation model was run on a set of 141 reference NTE events collected from 

a number of sources. Five engine manufacturers provided a total of 97 events; 10 reference NTE 

events came from each of the three engines tested in the lab during the transient testing; and 14 

events came from the pre-pilot CE-CERT data. The gaseous measurement allowance program 

used 195 reference NTE events, but 54 of the events were repeats in terms of speed and torque 

but with different gaseous emissions [1]. These events were removed and PM concentration 

estimations were generated for the remaining 141 events. Before and after errors were applied in 

the Monte Carlo simulation for each of these reference NTE events.

NTE brake-specific emissions results were calculated for PM using each of the three agreedupon NTE calculation methods. The three different BS emissions calculation methods referred 

to in this test plan are:

1. Method #1: Exhaust Flow-Torque-Speed Method

2. Method #2: Exhaust and Fuel Flow-Torque-Speed Method

3. Method #3: Fuel Flow--Torque-Speed Method

Table 5 lists the number of NTE events obtained from each data source and the BSPM emissions 

calculated using Method 1. The PM concentration used to calculate the Reference NTE PM 

emissions was developed generated by SwRI based on the Volvo engine used in this study. A 

simple model was developed to predict the PM concentration based on the speed, torque, and 

throttle position of the engine. The model was initially designed to match the actual PM 

emissions over a short transient cycle for which data had been recorded. These emissions have 

been computed with no error values added to the input parameters. For this report, emissions 

with no errors added will be labeled the “ideal” emissions. In contrast, the emissions with errors 

added through the Monte Carlo simulation will be labeled emissions “with errors”.

TABLE 5. REFERENCE NTE EVENTS AND IDEAL BSPM EMISSIONS

Source

Number 

of NTE 

Events

BSPM

g/hp-hr

Min Max

International 19 0.01146 0.03069

DDC 18 0.01303 0.03463
10

SwRI Project No. 14936.12

Caterpillar 20 0.01097 0.01789

Cummins 20 0.01561 0.03253

Volvo 20 0.01541 0.02701

Engine #1 10 0.01215 0.04066

Engine #2 10 0.01675 0.04872

Engine #3 10 0.01099 0.03669

Pre-Pilot 14 0.01375 0.03561

When the ideal brake-specific emission values were calculated for the various reference NTE 

events, it was noted that these ideal emission values were frequently different from one 

calculation method to another. While it was recognized that this was a realistic outcome, the 

Steering Committee felt that these discrepancies might introduce an unintended bias into the 

results of the Monte Carlo simulation. Therefore, the Steering Committee directed SwRI to 

adjust the NTE reference event data in order to align the brake-specific emission levels from all 

the calculation methods.

The adjustment was performed by first assuming that the Method 1 result was the desired value, 

and that the other two calculation methods would be aligned to that result. This meant that 

torque, speed, and exhaust flow values were not changed. The next step of the alignment process 

was to adjust CO2 values for the NTE event, in order to line up the Method 2 PM result with the 

Method 1 value. This was done by using a single multiplier on all CO2

The distribution of the ideal BSPM emissions data for the 141 reference NTE events was 

presented to the Steering Committee. It was noted that very few reference NTE events were at or 

below the BSPM threshold (0.02 g/hp-hr). Thus, the reference NTE events were adjusted to 

produce more events with ideal BSPM values below 0.02 g/hp-hr. The original and revised ideal 

BSPM distributions are depicted in 

values for the NTE event 

in question. Finally the fuel rate values and alignments were adjusted to bring Method 3 in line 

with Method 1 and 2. 

Figure 3. Note that the BSPM emission has data values 

spread above and below the corresponding NTE threshold. 
11

SwRI Project No. 14936.12

FIGURE 3. METHOD 1 IDEAL BSPM VALUES FOR REFERENCE NTE EVENTS

Table 6 provides a summary of some descriptive statistics for the reference NTE data set for 

BSPM emissions.

TABLE 6. DESCRIPTIVE STATISTICS FOR BSPM EMISSIONS FOR 141 

REFERENCE NTE EVENTS

Descriptive Statistic

BSPM

g/hp-hr

Minimum 0.010974

Maximum 0.048717

Mean 0.020753

Median 0.019101

Standard Deviation 0.006932
12

SwRI Project No. 14936.12

The parameter data provided in each reference NTE event was on a second-by-second basis with 

a minimum of 30 seconds and a maximum of 300 seconds. The input parameters required for the 

BSPM emissions calculation methods and the Monte Carlo simulation are listed in Table 7. An 

Excel file with a specific input format structure was used to standardize the format of the input 

files. Since the total hydrocarbons (THC) was selected as an input parameter, NMHC was 

computed as THC*0.98.
13

SwRI Project No. 14936.12

TABLE 7. INPUT PARAMETERS FOR REFERENCE NTE EVENTS

Variable 

Number

Input Variable Units Description

1 NTE Event Number integer All reference NTE events must be identified by an NTE number (e.g., 

001).

2 NTE Source alphanumeric The source of the NTE event is the company, organization and/or lab 

that created the event data.

3 Engine Make alphanumeric Engine Make

4 Engine Model alphanumeric Engine Model

5 Engine Displacement L Engine Displacement (L)

6 Date mm/dd/yyyy The day the NTE event data was created (mm/dd/yyyy).

7 Time Stamp hh:mm:ss.s Time in seconds. Each reference NTE must contain second-bysecond data only.

8 Wet CO2 % CO2 (%)

9 Wet CO % CO (%)

10 Wet kNO ppm NO (ppm) with intake air-humidity correction

11 Wet kNO2 ppm NO2 (ppm) with intake air-humidity correction

12 Wet THC ppm THC (ppm)

13 Exhaust Flow Rate scfm Exhaust flow rate (scfm)

To compute the % of PEMS flowmeter maximum flowrate we will 

need to know what size flowmeter was used for each NTE event.

Enter either 3, 4, or 5 to represent the following flowmeters and 

maximum flow rates:

3 = 3 inch EFM with maximum flow rate = 600 scfm

4 = 4 inch EFM with maximum flow rate = 1100 scfm

5 = 5 inch EFM with maximum flow rate = 1700 scfm

15 Speed rpm Engine speed (rpm)

16 Low Speed, nlo rpm To compute the % of normalized speed we will need nlo and nhi for 

the engine computed as follows:

nlo (rpm) = lowest speed below max power at which 50% max power 

occurs

18 Fuel Rate L/sec Fuel rate (L/sec)

To compute the % of maximum fuel rate we will need the max fuel 

rate of the engine for each NTE event.

Max fuel rate (L/sec)

20 Derived Torque N·m Torque (N·m)

21 Peak Torque N·m To compute the % of maximum torque we will need the peak torque 

of the engine for each NTE event

Peak torque (N·m)

22 Flow-weighted Average 

PM Concentration for 

Methods 1 & 2

μg/mol Flow-weighted average PM concentration, flow-weighted by the 

exhaust flow. Values were calculated based on a predictive model 

developed using transient and steady-state experimental data.

23 Flow-weighted Average 

PM Concentration for 

Method 3

μg/mol Flow-weighted average PM concentration, flow-weighted by the 

exhaust flow. Values were calculated based on a predictive model 

developed using transient and steady-state experimental data.

19 Max Fuel Rate L/sec

nhi (rpm) = highest speed above max power at which 70% max 

power occurs

14 Flowmeter Diameter 3, 4, or 5 (inches)

17 High Speed, nhi rpm
14

SwRI Project No. 14936.12

2.6 ERROR SURFACE GENERATION

During the discussions held at several Steering Committee meetings, 33 error surfaces were 

identified and considered for inclusion in the Monte Carlo simulation model. 25 of these error 

surfaces were the same surfaces used in the gaseous emissions in-use testing program. Of the 

remaining eight error surfaces, two were discarded (Delta PM EMI/RFI and Delta PM

Vibration). The six new error surfaces for the PM program were PM Steady-State (SS), PM 

Transient, PM Atmospheric Pressure, PM Ambient Temperature, Torque Engine Manufacturers 

and Fuel Rate Engine Manufacturers. This resulted in a final total of 31 error surfaces that were 

incorporated into the Model. These individual error surfaces encompassed a wide variety of 

error sources.

Table 8 lists the error surfaces examined during the study with the surfaces excluded by the 

Steering Committee designated in italics. All remaining ones were implemented in the 

simulation model. Each error surface was assigned a number for easy identification. 
15

SwRI Project No. 14936.12

TABLE 8. ERROR SURFACES FOR MONTE CARLO SIMULATION

For each of the measurement errors defined in Table 8, an error surface was created and used in 

the Monte Carlo simulation. Each error surface represented an additive error—or a subtractive 

error if the sign was negative—relative to the reference parameter value to which it was applied. 

Figure 4 through Figure 6 serve as a hypothetical example of how these error surfaces were

Component # Test Source Error Surface Description

1. Delta PM 1 Engine Dyno Delta PM SS AVL, Horiba and Sensors

2 Engine Dyno Delta PM Transient AVL, Horiba and Sensors

3 Environ Delta PM EMI/RFI Deleted by Steering Committee

4 Environ Delta PM Atmospheric Pressure AVL

5 Environ Delta PM Ambient Temperature AVL

6 Environ Delta PM Vibration Deleted by Steering Committee

2. Delta CO 7 Engine Dyno Delta CO SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

10 Environ Delta CO Atmospheric Pressure Same as Gaseous Study 

11 Environ Delta CO Ambient Temperature Same as Gaseous Study 

3. Delta NMHC 13 Engine Dyno Delta NMHC SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

 NMHC = 0.98*THC 14 Engine Dyno Delta NMHC Transient Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

16 Environ Delta NMHC Atmospheric Pressure Same as Gaseous Study 

17 Environ Delta NMHC Ambient Temperature Same as Gaseous Study 

19 Environ Delta Ambient NMHC Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

4. Delta Exhaust Flow 20 Engine Dyno Delta Exhaust Flow SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

21 Engine Dyno Delta Exhaust Flow Transient Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

22 Engine Dyno Delta Exhaust Flow Pulsation Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

23 Engine Dyno Delta Exhaust Flow Swirl Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

25 Environ Delta Exhaust EMI/RFI Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

27 Environ Delta Exhaust Temperature Same as Gaseous Study 

28 Environ Delta Exhaust Pressure Same as Gaseous Study 

5. Delta Torque 29 Engine Dyno Delta Dynamic Torque Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

30 Engine Dyno Delta Torque DOE Testing

(Interacting Parameters Test)

Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

31 Engine Dyno Delta Torque Warm-up

(Interacting Parameters Test)

Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

32 Engine Dyno Delta Torque Humidity/Fuel

(Independent Parameters Test)

Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

34 Engine Dyno Delta Torque Interpolation Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

35 Engine Manuf Delta Torque Engine Manuf New

6. Delta Fuel Rate 42 Engine Manuf Delta Fuel Engine Manuf New

7. Delta Speed 43 Engine Dyno Delta Dynamic Speed Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

8. Delta Fuel Rate 44 Engine Dyno Delta Dynamic Fuel Rate Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

9. Delta CO2

45 Engine Dyno Delta CO2 SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

46 Engine Dyno Delta CO2 Transient Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

49 Environ Delta CO2 Ambient Temperature Same as Gaseous Study 

Measurement Error Surfaces and Deltas Used in BSPM Calculations
16

SwRI Project No. 14936.12

created for every measurement error. Details on the construction of each error surface used in 

the simulation are provided in Section 2.6. The example illustrated in Figure 4 through Figure 6

represents the error surface for steady-state bias and precision PM concentration errors for an 

individual PEMS unit. The plots shown correspond to hypothetical PM emissions concentration 

data acquired in the laboratory. 

Figure 4

PEMS vs. Laboratory Nominal Results

was constructed from raw data acquired from steady-state engine lab tests with the 

PEMS at repeat testing at various concentration levels (PM μg/mol). The plot pools all bias and 

precision errors for the PEMS tested for all steady-state modes. A nominal target of 1010 repeat 

measurements of PM

Figure 4

were taken on each PEMS unit for each value of the corresponding average 

lab PM values (i.e., lab nominal value). The 110 PEMS measurements were plotted against the 

corresponding measurements using laboratory equipment. Shown in are the 5th, 50th, 

and 95th percentiles corresponding to the distribution of these 110 observations using the PEMS 

at each average PM concentration level (note that the distribution of data at each PM level may 

not represent a normal distribution). Since the 50th

Figure 4

percentiles do not lie on the dashed 

(diagonal) line of perfect agreement, the data suggest that there is a bias error between the PEMS 

and lab results. In essence this graph summarizes the statistical distribution measured by the 

PEMS at each concentration level sampled. The example plot in shows only 6 discrete 

average PM concentration levels (ranging from 10-60 μg/mol). However, the actual number of 

discrete concentration levels was determined using the total number of operating conditions 

actually run for all the tests on the engine. In the section on Steady-State Repeat Engine Testing 

and Error Surfaces it is reported that 66 operating modes conditions from an initial number of 

8080 operating conditions were selected for construction of the steady-state PM error surface. 

From these 6 operating modes several discrete PM concentration levels were defined which were

used in the error surface plots for the Monte Carlo simulation.6.
17

SwRI Project No. 14936.12

FIGURE 4. ERROR SURFACE CONSTRUCTION: PEMS VS. LABORATORY 

RESULTS

Figure 5

(PEMS – Laboratory) Deltas vs. Lab

illustrates the “error band” measured during testing. This plot was created by first 

subtracting the individual “lab nominal” PM value from the corresponding individual PEMS PM

measurement for each test run. This difference was defined as the “delta” error. Second, these 

“PEMS - Laboratory” delta errors were pooled at each average lab nominal PM value to obtain 

the 5th, 50th, and 95th percentile values displayed in Figure 5. Therefore, the plot represents the 

average PM lab nominal at 6 discrete concentration levels versus the percentiles of the delta 

errors computed from the PEMS and laboratory individual test results. 

Error Surface for SS PM Concentration

18

26

38

45

56

69

13

24

36

42

53

64

8

19

33

39

51

61

0

10

20

30

40

50

60

70

80

0 10 20 30 40 50 60 70 80

PM μg/mole (lab,mean)

PM μg/mole (PEMS)

diagonal 95th percentile

50th percentile (median) 5th percentile
18

SwRI Project No. 14936.12

FIGURE 5. ERROR SURFACE CONSTRUCTION: (PEMS - LAB) VS. LABORATORY 

RESULTS

This step normalized the plot in 

Variability Index vs. (PEMS – Laboratory) Deltas and Lab Nominal

Figure 5 using what is called a “variability index (ic)”. This 

index represented the value randomly drawn by the Monte Carlo simulation in order to select a 

given error level. For the 5th and 95th percentile of the truncated normal it was allowed to vary 

from –1 to +1, respectively. The likelihood of “ic” being any value between –1 through +1 was 

specified by a “probability density function (PDF)” assigned to ic. In the case of this example, ic. 

was assumed to vary according to a standard normal (i.e, bell-shaped) distribution during the 

Monte Carlo simulations. This was because it was believed that the distribution of PM errors 

due to steady-state bias and precision would be centered about the 50th

Figure 5

percentile of the full 

range of conditions measured. Each set of data for each lab “setpoint” average (i.e., lab nominal 

value) in was normalized by aligning the corresponding 5th Figure 

5

percentile error from 

with ic = -1, the 50th percentile error with ic = 0, and the 95th percentile error with ic 

Error Surface for SS PM Concentration

8

6

8

5

6

9

3

4

6

2

3

4

-2

-1

3

-1

1 1

-4

-2

0

2

4

6

8

10

0 10 20 30 40 50 60 70

PM μg/mole (lab,mean)

PM μg/mole (PEMS) - PM μg/mole (lab)

95th percentile 50th percentile (median) 5th percentile

= +1. 
19

SwRI Project No. 14936.12

These values were then plotted in Figure 6, where the y-axis is the variability index, the x-axis is 

the average lab nominal PM value, and the z-axis is the delta PM value. Notice that, when using 

this normalization approach, the 5th, 50th, and 95th

Figure 5

percentile values remain equivalent between 

and Figure 6. This development of the error surfaces from the lab data was the 

procedure used in the gaseous emissions in-use testing program.

FIGURE 6. ERROR SURFACE CONSTRUCTION: ERROR AT VARIABILITY INDEX 

FOR 5TH AND 95TH

For the PM measurement allowance program it was decided by the Steering Committee to 

expand the tails of the truncated normal distribution in the error surface formation in order to 

allow larger PM deltas to be sampled during the Monte Carlo simulation. Instead of truncating 

the lower tail at the 5

PERCENTILES VS. LABORATORY RESULTS

th percentile, it was moved to the 1st percentile. Likewise on the upper tail, 

the 95th percentile was moved to the 99th

8 6 8 6 9

3 4 6 2 3 4

-2 -1 3 -1 1 1

-1.5

-1

-0.5

0

0.5

1

1.5

0 10 20 30 40 50 60 70

ic_SS_PM

PM μg/mole (lab,mean)

Error Surface for SS PM Concentration

95th percentile 50th percentile (median) 5th percentile

Error Surface: z-axis = ΔSS_PM_μg/mole

5

percentile. Since the original truncated normal 

distribution was defined by a mean = 0 and a standard deviation = 0.60795, the resulting indices 
20

SwRI Project No. 14936.12

corresponding to the 1st and 99th

Figure 7

percentiles are -1.4143 and +1.1413, respectively, as illustrated 

in .

FIGURE 7. TRUNCATED NORMAL DISTRIBUTION PERCENTILES

Finally, the 5th and 95th percentile (PEMS – LAB) delta values at each lab nominal value were 

used to compute the corresponding values of the truncated normal for the 1st and 99th percentiles. 

To redefine the error surface delta at the 1st percentile, the standard deviation for the normal 

distribution below the 50th

Using the mean = 50

percentile is defined as follows:

DD DD 1 = DD VV 50 h − DD VV 5 h

1.6449

th percentile delta and the standard deviation computed above, the 1st

-3 -2 -1 0 1 2 3

Monte Carlo Truncated Normal Sampling Distribution

N(0,0.60795)

-1.4143

Area= 0.05

Area = 0.01

-1.0 1.0

5th 95th

1.4143

Percentiles→ 1st 99th

percentile can be found using the Excel NORMINV function: NORMINV(0.01,mean,standard 
21

SwRI Project No. 14936.12

deviation1st). Similarly, to redefine the error surface delta at the 99th percentile, the standard 

deviation for the normal distribution above the 50th

Using the mean = 50

percentile is defined as follows:

DD DD 99 h = DD VV 95 h − DD VV 50 h

1.6449

th percentile delta and the standard deviation computed above, the 99th

percentile can be found using the Excel NORMINV function: NORMINV(0.99,mean,standard 

deviation99th). Taking the data for the error surface in Figure 6 and redefining it to include the 1st

and 99th percentile truncated normal values results in the error surface displayed in Figure 8. All 

the error surfaces carried over from the gaseous program that were sampled using a truncated 

normal were redefined at the 1st and 99th percentiles. Those error surfaces that were sampling 

using the uniform distribution remained unchanged. Error surfaces such as the one presented in 

Figure 6 are the error deltas the Monte Carlo simulation program used during calculation of the 

BSPM emissions “with errors”.
22

SwRI Project No. 14936.12

FIGURE 8. ERROR SURFACE CONSTRUCTION: ERROR AT VARIABILITY INDEX 

FOR 1ST AND 99TH PERCENTILES VS. LABORATORY RESULTS

2.7 ERROR SURFACE SAMPLING AND INTERPOLATION

The error model used two different probability density functions to sample the error surfaces,

depending upon which experimental parameter the surface represented. To sample error surfaces 

that were generated from the lab test results (Section on Engine Dynamometer Laboratory 

Testing), and the applicable environmental test results , the model used a truncated standard 

normal PDF because these tests were designed to evenly cover the full, but finite, range of 

engine operation and ambient conditions. To sample error surfaces that were generated from the 

pressure and temperature environmental test results (Section on Environmental Chamber 

Testing), the model used a uniform PDF because these tests were already designed to cover the 

typical range and frequency of the respective conditions. Both of these sampling distributions 

are depicted in Figure 9.

10.1 6.8 8.8 7.2 11.1

3 4 6 2 3 4

-4.1 -3.1 1.8 -2.2 0.2 -0.2

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 10 20 30 40 50 60 70

ic_SS_PM

PM μg/mole (lab,mean)

Error Surface for SS PM Concentration

99th percentile 50th percentile (median) 1st percentile

Error Surface: z-axis = ΔSS_PM_μg/mole

6.2
23

SwRI Project No. 14936.12

FIGURE 9. TRUNCATED STANDARD NORMAL AT 1ST AND 99TH

When using the truncated standard normal PDF (see 

PERCENTILES 

AND UNIFORM PROBABILITY DENSITY FUNCTIONS

Figure 9), the Monte Carlo model sampled 

normal deviates that ranged between -1.4143 and +1.4143. These were used as the ic values 

defined in the section on Error Surfaces. Similarly, the pressure and temperature environmental 

tests used a uniform PDF to sample test time, from which calculated errors were used. All 

temperature error surfaces related to the four emissions were sampled uniformly from 1 to 1080 

minutes while the error surfaces related to the pressure were sampled uniformly from 1 to 720 

minutes. Exhaust flow error surface for temperature was sampled uniformly from 1 to 478 

minutes while the exhaust flow for pressure was sampled uniformly from 1 to 360 minutes. The 

errors from all the other tests were aligned with the truncated standard normal PDF such that 

each of the 50th percentile error values at each of the tested signal magnitudes was centered at the 

median (i.e., 0 value) of the PDF, and the 1st and 99th percentile error values at each of the tested 

signal magnitudes were aligned with the extreme negative (ic = -1.4143) and positive (ic

Each error surface was sampled along its i

= 

+1.4143) edges of the PDF, respectively.

c axis (y-axis) once per trial for a reference NTE event

simulation. Hence, every error surface had a separate randomly selected ic for each trial. Since 

each reference NTE event contained second-by-second parameter data, except for PM for the 

Sensors PPMD and Horiba TRPM PEMS, the error surface was sampled at a given ic on the yaxis and at the several selected parameter values on the x-axis that corresponded to each second 

Probability Density Functions for Sampling Error Surfaces Once Per NTE Event 

-1.41

0.00

1.41

5 4 3 2 1 0

Relative Probability

i c

Lab Tests, Normal, SD=0.60795, truncate @ -1.4143 & 1.4143

Environmental Tests, Uniform

Note: A non-truncated normal 

distribution with SD=0.60795 has P 

values of 0.01 and 0.99 at i c =-1.4143 and 

ic=+1,4143 respectively.
24

SwRI Project No. 14936.12

of the reference NTE event. The sampled error value was determined for the given second and 

parameter along the error axis (z-axis) at the intersection of the ic value and the parameter value 

from the reference NTE event. This was accomplished by taking each second in the reference 

NTE event and finding the two adjacent x-axis values from the error surface between which to 

linearly interpolate to obtain the error surface x-value. Each second in the reference NTE event 

was linearly interpolated with the same ic

Figure 10

value for a particular trial at the error surface x-value. 

If any of the sampled lab nominal values (PM, NMHC, CO, Speed, Fuel Rate, etc.) exceeded the 

upper or lower limits of the parameter error surface, the value of the closest endpoint of the error 

surface was assigned to them.

depicts an example of the error surface sampling using a steady-state PM error surface 

containing 10 lab nominal PM x-axis values. For this particular trial, the randomly selected ic is 

-0.5. The example reference NTE event is noted by the symbol ‘*’ and it plotted at ic = -0.5 for 

each second in the NTE event.
25

SwRI Project No. 14936.12

FIGURE 10. STEADY-STATE PM ERROR SURFACE FOR AVL WITH EXAMPLE 

SAMPLING FOR A REFERENCE NTE EVENT

2.8 BRAKE-SPECIFIC EMISSIONS CALCULATIONS

Errors from Sections 4, 5, and 6 were combined by adding all of the sampled errors once per 

trial for each reference NTE event simulation. For example, in order to assess the errors in PM

concentration by calculation Method #1, several error surfaces were sampled and added to the 

corresponding parameter in the Method #1 calculation and the resulting BSPM “with errors” was 

computed. The errors used in this calculation for the Horiba and Sensors are the following (note

that the corresponding error surface numbers are provided in the subscripts):

PM μg/mol ‘with errors’ = PM μg/mol_reference + ∆ PM μg/mol_1 + 

∆ PM μg/mol _2 

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 50 100 150 200 250 300 350 400 450

ic_SS_AVL

PM μg/mol (lab,nom)

SS Error Surface for AVL PM Concentration

99th percentile 50th percentile (median) 1st percentile

ic = -0.5 NTE Event

Error Surface: z-axis = ΔSS_PM_AVL_μgmol
26

SwRI Project No. 14936.12

Exhaust Flow % ‘with errors’ = Exhaust Flow % reference + 

∆ Exhaust Flow % 20 ∆ Exhaust Flow_% 21 + 

∆ Exhaust Flow % 22 + ∆ Exhaust Flow % 23 + 

∆ Exhaust Flow % 25 + ∆ Exhaust Flow % 27 + 

∆ Exhaust Flow % 28 

Torque % ‘with errors’ = Torque % reference + 

∆ Torque % 29 + ∆ Torque % 30 + 

∆ Torque % 31 + ∆ Torque % 32 + 

∆ Torque % 34 + ∆ Torque % 35 

Speed % ‘with errors’ = Speed % reference + ∆ Speed % 43 

where,

∆ 1,2 = PM concentration errors due to steady-state and transient errors,

∆ 20,21 = exhaust flow errors due to steady-state and transient errors,

∆ 22,23 = exhaust flow errors due to pulsation and swirl,

∆ 25 = exhaust flow errors due to ambient temperature,

∆ 27,28 = exhaust flow errors due to temperature and pressure,

∆ 29 = torque errors due to dynamic torque,

∆ 30,31 = torque errors due to DOE and warm-up,

∆ 32 = torque errors due to interacting parameters humidity and fuel,

∆ 34,35 = torque errors due to interpolation and engine manufacturers,
27

SwRI Project No. 14936.12

∆ 43

Using the formulas for the calculation methods, the BSPM for Method #1 was computed without 

errors (“ideal”) and then with all the errors applied as outlined above. 

= speed errors due to dynamic speed

Table 9 lists all error 

surfaces used by each calculation method for the PM emissions.

TABLE 9. ERROR SURFACES USED FOR COMPUTING BRAKE-SPECIFIC PM 

EMISSIONS BY THREE CALCULATION METHODS

Component # Error Surface Method 1 Method 2 Method 3

1. Delta PM 1 Delta PM SS   

2 Delta PM Transient   

4 Delta PM Atmospheric Pressure   

5 Delta PM Ambient Temperature   

2. Delta CO 7 Delta CO SS  

10 Delta CO Atmospheric Pressure  

11 Delta CO Ambient Temperature  

3. Delta NMHC 13 Delta NMHC SS  

 NMHC = 0.98*THC 14 Delta NMHC Transient  

16 Delta NMHC Atmospheric Pressure  

17 Delta NMHC Ambient Temperature  

19 Delta Ambient NMHC  

4. Delta Exhaust Flow 20 Delta Exhaust Flow SS  

21 Delta Exhaust Flow Transient  

22 Delta Exhaust Flow Pulsation  

23 Delta Exhaust Flow Swirl  

25 Delta Exhaust EMI/RFI  

27 Delta Exhaust Temperature  

28 Delta Exhaust Pressure  

5. Delta Torque 29 Delta Dynamic Torque  

30 Delta Torque DOE Testing

(Interacting Parameters Test)  

31 Delta Torque Warm-up

(Interacting Parameters Test)  

32 Delta Torque Humidity/Fuel

(Independent Parameters Test)  

34 Delta Torque Interpolation  

35 Delta Torque Engine Manuf  

6. Delta Fuel Rate 42 Delta Fuel Engine Manuf 

7. Delta Speed 43 Delta Dynamic Speed  

8. Delta Fuel Rate 44 Delta Dynamic Fuel Rate 

9. Delta CO2 45 Delta CO2 SS  

46 Delta CO2 Transient  

49 Delta CO2 Ambient Temperature  
28

SwRI Project No. 14936.12

2.9 CONVERGENCE AND NUMBER OF TRIALS

Since the Test Plan did not include a provision for convergence criteria, the Steering Committee 

was tasked to develop a convergence method. The main goal was to define how many 

simulation trials at a given reference NTE event were required to estimate the 95th percentile 

BSPM emission differences with a given precision. Although the Crystal Ball software 

contained precision control options, the method used to compute a confidence interval on 

percentiles was based on an analytical bootstrapping method which was not adequately 

documented. Thus, an independent convergence method was proposed and accepted by the 

Steering Committee.

A nonparametric statistical technique [3] was proposed which defined a 90% confidence 

interval for the 95th

1. Run the Monte Carlo simulation for N trials.

percentile of the BSPM emissions differences for an individual reference 

NTE simulation. If the width of the 90% confidence interval was less than 1% of the BSPM 

emissions threshold, then convergence was met. The following steps define the convergence 

method:

2. Order the BS emissions differences from smallest to largest.

3. Identify the trial number at the lower end of the 90% confidence interval

nlower = 0.95 * N −1.645 0.95 * 0.05 * N

4. Identify the trial number at the upper end of the 90% confidence interval

nupper = 0.95 * N + 1.645 0.95 * 0.05 * N

5. Compute (BSPM difference value at nupper) – (BSPM difference value at nlower

6. If the result in (5) < 1% of the BSPM emissions NTE threshold then convergence is met.

).

7. The BSPM threshold was defined as 0.02 g/hp-hr. Thus, 1% of the threshold was 0.0002 

g/hp-hr.

The Screening Committee agreed to the proposed convergence criteria outlined above. During 

the initial simulation runs for 20 reference NTE events, convergence was not met at the 1 percent 

criteria level until 60,000 trials were run. This only applied to the AVL PEMS unit at each of the 

three calculation methods. The Horiba and Sensors units only reached convergence at the 1 

percent criteria for approximately half of the 20 reference NTE events simulations. Upon 

examination of the distributions of the delta BSPM emissions generated from the simulations, 

some of the distributions were positively skewed which would make convergence very difficult 

at the 1 percent level. This information was presented to the Steering Committee wherein a 
29

SwRI Project No. 14936.12

decision was made to relax the convergence level to 2 percent or higher, depending on the 

outcomes of the simulations. 

In summary, the 141 reference NTE events were run at 40,000 trials and convergence was 

checked. If the width of the confidence interval on the 95th

2.10 SIMULATION OUTPUT

percentile delta BSPM emission was 

approaching 2 percent of the threshold, then the simulation was continued for up to 65,000 trials.

During the simulation of a reference NTE event, differences between the BSPM emissions “with 

errors” and the ideal BSPM emissions were obtained by each of the three PEMS model units and 

each of the three applicable calculation methods. These differences were computed thousands of 

times (once per trial) until the model converged. Then the 95th percentile difference value was 

determined for each reference NTE event’s distributions of BS differences for the PM emissions

for all three PEMS units and applicable calculation methods.

The output from the Crystal Ball simulation for each reference NTE event was saved in two 

separate Excel files: an EXTRACT and a REPORT file. The EXTRACT file contained 

descriptive statistics on all differences computed for BSPM emissions by all three calculation 

methods, percentiles (0%, 5%, 10%,...95%, 100%) of the differences in BSPM emissions, 

sensitivity data for all error surfaces, and differences in BSPM emissions computed at each trial 

in the simulation.

The REPORT file contained a summary of the differences in the BSPM emissions for all three 

PEMS units and applicable calculation methods including descriptive statistics, the number of 

trials, a frequency histogram of the differences in BSPM emissions, and percentiles (0%, 5%, 

10%,...95%, 100%) of the differences in BSPM emissions. Also included were descriptive 

statistics on each ic

2.11 STEP-BY-STEP SIMULATION EXAMPLE

distribution sampled for each error surface. Lastly, sensitivity charts for the 

differences in BSPM emissions for the three PEMS unit and applicable calculation methods were 

stored. These charts provided information on how much each error surface influenced the 

differences computed between the BSPM emissions “with errors” and the ideal BSPM 

emissions.

A more detailed description of the Crystal Ball output files can be found in Appendix B.

In order to clarify the simulation process the following step-by-step summary is provided. This 

example assumes that a single reference NTE event was simulated for the BSPM difference 

computations. Figure 11 provides an overview of the simulation process.
30

SwRI Project No. 14936.12

FIGURE 11. OVERVIEW OF MONTE CARLO SIMULATION FOR BSPM

STEP 1 Enter the reference NTE input parameters into the Monte Carlo (MC) simulation 

model. These include the emissions concentrations, exhaust flow, torque, speed and fuel rate 

data used in all three calculation methods.

STEP 2 Compute the “ideal” BSPM by all three PEMS model unit and applicable 

calculation methods from the reference NTE event.

STEP 3 Set-up the Monte Carlo simulation parameters in Crystal Ball. An Excel 

spreadsheet model was developed for use with Crystal Ball MC software for error analysis of 

brake-specific emissions. Crystal Ball is a graphically-oriented forecasting and simulation 

software that runs on Microsoft Windows and Excel. The simulations run in this program used 

Crystal Ball Version 11.1.1 and were run on PCs configured with a Pentium 4 CPU, 3.39 GHz, 

3.50 GB RAM, 232 GB hard drive and Windows XP operating system. Microsoft Excel 2003 

SP was the spreadsheet software.
31

SwRI Project No. 14936.12

The options exercised in running Crystal Ball included the following:

• Number of trials = 40,000 

o If convergence was not met at the 2% criteria then # trials = 65,000

• Monte Carlo sampling method with random initial seeds

• Normal speed run mode

• Suppress chart windows (fastest run time)

The Excel spreadsheet is in a modular structure following the specified model outline, and it 

makes provisions for the three identified calculation modules. Input cells to the model are 

clearly identified to facilitate any revisions that may become necessary for users who want to 

exercise the model with other Monte Carlo software such as @Risk or newer versions of Crystal 

Ball. The spreadsheet was tested with controlled test cases of simplified input distributions with 

the Crystal Ball add-on to confirm correct model implementation in accordance with this test 

plan. At least one typical analysis was run as an additional confirmation, and two independent 

checks were made on the ideal emissions by other SwRI staff. A complete description of the 

spreadsheet computations is contained in Appendix C 

STEP 4 Execute a single MC trial by randomly generating a separate ic for each error 

surface used in the three calculations.

STEP 5 For each second in the reference NTE event, interpolate the ∆ error for all error 

surfaces at the input parameter values and the randomly generated ic. Figure 12 illustrates all the 

error surfaces available and where the corresponding ∆ errors are added. The numbers in 

parentheses represent the error surface number in the Monte Carlo simulation.
32

SwRI Project No. 14936.12

FIGURE 12. ERROR SURFACES INCLUDED IN MONTE CARLO SIMULATION

STEP 6 Compute one BSPM “with errors” for the given MC trial by adding all the ∆ error 

values to the reference NTE data and then calculating the BSPM by all three PEMS units and 

applicable calculation methods.

STEP 7 Compute BSPM difference for the current trial:

BSPM emission “with errors” – “Ideal” BSPM emission

STEP 8

Environ 

chambers

Engine

dyno

Environ 

chambers

Engine 

dyno

PM

∆PMSS_ppm (1)

∆PMTR_ppm (2)

∆PMpress_ppm (4)

∆PMtemp_ppm (5)

Εxhflow

∆ExhflowSS (20)

∆ExhflowTR (21)

∆Exhflowpulsation (22)

∆Exhflowswirl (23)

∆ExhflowEMI/RFI (25)

∆Exhflowtemp (27)

∆Exhflowpressure (28)

Speed ∆SpeedDynamic (43)

∆FuelRateDynamic Fuel Rate (44)

Torque

∆TorqueDynamic (29)

∆Torquewarmup (31)

∆Torquehumidity (32)

∆Torqueengine-manuf (35)

Engine dyno

∆FuelRateengine-manuf (42)

Engine 

dyno

CO

∆COSS_ppm (7)

∆COpress_ppm (10)

∆COtemp_ppm (11)

Engine 

dyno

NMHC

∆NMHCSS_ppm (13)

∆NMHCTR_ppm (14)

∆NMHCpress_ppm (16)

∆NMHCtemp_ppm (17)

∆NMHCambient (19) Engine 

dyno

CO2

∆CO2SS_ppm (45)

∆CO2TR_ppm (46)

∆CO2temp_ppm (49)

Environ 

chambers

Environ 

chambers

∆TorqueDOE (30)

∆Torqueinterpolation(34)

Repeat Steps 4-7 until the number of trials is met.
33

SwRI Project No. 14936.12

STEP 9 Check the differences in BSPM for all three PEMS units and applicable 

calculation methods to be certain that the convergence criteria are met. If convergence is met for 

all three calculation methods, continue to Step 10. Otherwise, return to Step 4 and run the Monte 

Carlo simulation for an additional 25,000 trials until the total number of trials is 65,000.

STEP 10 Select the 95th percentile from the distribution of BSPM differences for each of 

the three PEMS units and applicable calculation methods. Store the ideal BSPM and the 95th

percentile BSPM differences for computing the measurement allowance.

STEP 11

2.12 MEASUREMENT ALLOWANCE GENERATION

Repeat Steps 1-10 for each reference NTE event.

The generation of a set of measurement allowances represented the final outcome of this 

program. The Test Plan provided a methodology by which all of the data from the millions of 

Model simulation runs would be collected and analyzed statistically, in order to generate a set of 

potential measurement allowances for each PEMS model unit, one for each of the calculation 

methods. The Test Plan then outlined a specific method by which the final set of allowances 

would be chosen from among deltas generated for each of the three calculation methods. The 

assumption made by the Test Plan, was that the final outcome of all previous efforts would be a 

set of validated potential measurement allowance values for each PEMS unit. Each potential 

allowance was expressed as a percentage of its associated BSPM NTE threshold. These 

measurement allowances were computed by a regression method or a median method as 

described below.

2.12.1

This method involved determining the correlation between the 95

Regression Method

th percentile differences versus 

the ideal emission values for the reference NTE dataset. For each combination of PEMS units 

and calculation method, a least squares linear regression of the 95th percentile differences versus 

the ideal emissions results was computed. If the R2 value from the regression model was greater 

than 0.85 and the SEE (standard error of the estimate or root-mean-squared-error) was less than 5 

percent of the median ideal BS emissions, then the linear regression equation was used to 

determine the measurement allowance for that PEMS unit and calculation method. To 

determine the measurement allowance the NTE threshold was used to predict the measurement 

allowance from the regression model. The measurement allowance was then expressed as a 

percentage of the NTE BSPM threshold value (0.02 g/hp-hr).
34

SwRI Project No. 14936.12

2.12.2

If the linear regression did not pass the aforementioned criteria for the R

Median Method

2 and SEE statistics, then 

the median value of the 95th percentile differences from the 141 reference NTE events was used 

as the single measurement allowance for a combination of emissions and calculation method. 

The measurement allowance was then expressed as a percentage of the NTE threshold value.

After all 95th

The anticipated outcome from the model runs, analysis, and validation efforts can be represented 

as a table similar to the one shown in 

percentile distributions were evaluated, there were seven measurement allowances 

corresponding to the combinations of the three PEMS units and the applicable calculation 

methods. 

Next, the calculation method with the minimum normalized PM value will be chosen and the 

corresponding normalized PM value will be selected as the best measurement allowance for PM, 

assuming it validates. This PM measurement allowance would be the very last value added to 

the actual brake-specific NTE PM threshold for a given engine, based on actual family emissions 

limit, mileage, model year, etc. Note that if any measurement allowance was determined to have 

a value less than zero, then that measurement allowance was set equal to zero.

The BSPM NTE threshold used for this program was 0.02 g/hp-hr. This NTE threshold was 

determined by EPA and approved by the Steering Committee during the generation of the Test 

Plan.

These threshold values are of critical importance to the program, as they provide the basis for the 

scaling of measurement allowances, the assessment of model convergence, and a variety of other 

calculations performed during this program. The general philosophy of the Test Plan was to 

determine measurement allowances based on errors at these emission levels, especially in the 

case of any errors that scaled with emission level.

Table 10. The table illustrates both the model outcome, 

and the process for selecting the final measurement allowance values for the AVL PEMS.
35

SwRI Project No. 14936.12

TABLE 10. EXAMPLE OF SELECTION OF MEASUREMENT ALLOWANCE AT 0.02 

G/HP-HR NTE THRESHOLD FOR THE AVL PEMS

Allowance at Respective NTE threshold (%)

Calc. Method 

Method 1

Exhaust Flow 

Torque-Speed

Method 2

Exhaust and Fuel 

Flow TorqueSpeed

Method 3

Fuel Flow TorqueSpeed

BSPM 38 % 18 % 20 %

Selected Method  Exhaust and Fuel Flow Torque-Speed Method

The intent of the final selection process was to choose the smallest of the three normalized PM 

values for the final measurement allowance. At that point, the percentages given for the chosen 

calculation method would be applied to the BSPM NTE threshold value in order to generate the 

final additive, brake-specific measurement allowances.

An implicit assumption of the process, as described in the Test Plan, was that the values 

produced by the model for all three calculation methods would be successfully validated. In the 

event that this did not occur, it would be necessary for the Steering Committee to determine a 

valid alternate course of action, in order to determine the final measurement allowance values.

The final model run and the selection and generation of measurement allowances are described 

fully in Section 6.2 of this report, including the final allowances approved by the Steering 

Committee.

2.13 MODEL VALIDATION

For reasons discussed earlier, the measurement allowances were generated using a Monte Carlo 

computer model. As with all simulations, it is vital that such a model be validated through 

comparison with real experimental data. In this case, the Measurement Allowance model needed 

to be validated against a data set generated through actual in-use field testing. Because the 

model generates an incremental error in comparison to a Laboratory Reference, a suitable in-use 

reference measurement was needed for comparison to the PEMS measurements. The Steering 

Committee determined that the CE-CERT Mobile Emission Laboratory, operated by the 

University of California-Riverside, would be an appropriate reference for validation of the 

model-based in-field testing.
36

SwRI Project No. 14936.12

To ensure that the validation was not disturbed by some inherent bias between the SwRI 

Reference Laboratory and the CE-CERT MEL validation reference, a correlation exercise was 

performed between the two laboratories, prior to the start of on-road validation efforts. The CECERT MEL was brought to SwRI’s laboratory facilities in San Antonio, Texas, and a side-byside correlation test was run. During this test, exhaust from the same test engine was alternately 

routed to the measurements systems of both SwRI and CE-CERT. This was done repeatedly 

over the course of three days of testing. The data was then supplied to the Steering Committee, 

in order to allow for a determination to be made that correlation between the facilities was 

acceptable for the purposes of validation of the model. 

After the correlation exercise was completed, a 2007 test truck with a Cummins engine was 

procured by CE-CERT for use in this validation exercise. In addition, two Sensors PEMS used 

at SwRI during the program were also delivered to CE-CERT. A third PEMS unit of the same 

type was provided by Sensors. The steering committee allowed Sensors to provide a similar 

model with some small hardware upgrades for testing. CE-CERT then conducted a series of onroad test runs over various driving routes in California, which were designed to take the test 

truck through a wide range of environmental and ambient conditions. During these tests, 

simultaneous measurements were made with the PEMS and the MEL in order to generate a 

validation data set. This formed the primary validation set for the model.

Because the CE-CERT MEL does not readily incorporate a means of direct torque measurement 

on a vehicle, the on-road validation data set could not be used to validate model errors associated 

with broadcast torque. 

The difference between the PEMS results and the CE-CERT trailer results will be compared to 

the error predicted by the Monte Carlo model. To validate the Monte Carlo model, data must be 

run through the model and the model results must predict the actual test results within a 

reasonable level of accuracy.

Validation will be based on the following procedure. For each reference NTE event, the Monte 

Carlo model will be used to generate the 5th and 95th percentiles of the simulated distribution of 

the brake-specific PM emission differences. In order to obtain simulations representing similar 

conditions to those obtained on-road, some error surfaces may need to be suppressed in the 

simulations since not all of them may be applicable to the on-road conditions. The choice of 

which error surfaces to suppress would need to be made by the Steering Committee. 

Next, the 5th and 95th delta percentiles obtained from the above simulations will be separately fit 

to a line or curve using two chosen methods: a linear regression procedure and a local regression 

(loess) technique [4]. Depending on which of the resulting two fits is best for each set of data 

(i.e., either for the 5th percentile deltas or the 95th percentile deltas), the resulting line or curve 

will be used as one of the lower or upper limits for the on-road data.
37

SwRI Project No. 14936.12

To determine the best fit for a given set of delta percentiles (i.e., 5th or 95th), a simple regression 

line initially will be fit to the data. If a least squares linear regression of the 5th or 95th percentile 

deltas versus the ideal PM emission has an r

2 greater than 0.85 and an SEE less than 5 percent of 

the median ideal PM emissions, then the regression line will be used. If this set of criteria is not 

met, then a loess fit will be used. Since a loess regression requires the selection of a smoothing 

parameter [5] to smooth the data, the chosen smoothness parameter should balance the residual 

sum of squares against the smoothness of the fit. 

The on-road delta errors, obtained from the results of collecting data on several NTE events 

during on-road operations, will be plotted on a graph containing the 5th and 95th

A hypothetical example of a validation plot is given in 

percentile delta 

limits determined from the regression fits chosen above. The graph will consist of a plot of delta 

PM versus ideal PM. The number of on-road points outside these limits will be determined and 

expressed as a percentage of the total number on on-road data points. If this number does not 

exceed 10% of the total number of on-road data, the simulation data will be considered to be 

valid.

Figure 13 and Figure 14. The plots 

shown correspond to gaseous emissions concentration data that were collected in the prior PEMS 

study. Figure 13 contains the 5th and 95th validation limits for NOX

Figure 14

data determined by fitting a 

linear regression model to the simulated data for both limits. contains the 5th and 95th

validation limits for NOX data determined by fitting a loess model to the simulated data for both 

limits. 
38

SwRI Project No. 14936.12

FIGURE 13. LINEAR REGRESSION FIT TO 5TH AND 95TH PERCENTILE DELTAS
39

SwRI Project No. 14936.12

FIGURE 14. LOESS REGRESSION FIT TO 5TH AND 95TH PERCENTILE DELTAS

Validation of the model was assessed independently for the PM pollutant for each of the three 

PEMS model units, and for each of the applicable three calculation methods. A full description 

of the validation efforts, including the data analysis methodology and the results of PM 

validation for each PEMS unit by all three calculation methods is given in Section 2.4,with the 

exception of the CE-CERT on road validation testing. 

3 PART 1065 PEMS AND LABORATORY AUDIT

3.1 1065 PEMS AUDIT

Before the start of engine testing, each PEMS was required to pass the verifications set forth in 

CFR Part 1065. Because the measurement of PM is a non-standard process there are no audits 

specified on the actual PM measurement, but the instruments still needed to pass the necessary 

flow, temperature, and pressure audits. Table 11 lists the Part 1065 audits performed on the lab 

and on the PEMS.
40

SwRI Project No. 14936.12

TABLE 11. SUMMARY OF PART 1065 AUDITS

Description CFR Reference Lab PEMS

Linearity 1065.307 x1 x2

Torque Meter 1065.310 x

Pressure, Temperature, Dewpoint 1065.315 x

Fuel Flow 1065.320 x

Intake Flow 1065.325 x

CVS Verification 1065.341 x

PM Balance Verification 1065.390 x

1 Linearity for lab performed on flow meters, torque meter, pressures, and temperatures

2 Linearity for PEMS performed on flow meters

Since no analyzer verifications were performed, the most critical audits on the PEMS were the 

linearity verifications of flow measurements. Because all three types of PEMS use flow 

measurements to calculate a dilution ratio, the accuracy of the flow measurements is directly 

related to the accuracy of the reported PM emissions. The AVL and Horiba PEMS measure their 

total and dilution flow to calculate a dilution ratio while the Sensors PEMS measures the dilution 

and sample flow. The following equation is used to calculate the dilution ratio:

DD RR = TT FFFF FF

SS FFFF FF = TT FFFF FF

TT FFFF FF − DD FFFF FF

3.1.1 Horiba Flow Audits

The Horiba PEMS has four flow measurements in the system: dilution flow, DCS flow, make-up 

air flow, and total flow. During typical operation all flows but the dilution flow are held constant. 

The dilution flow rate is varied to sample proportionally from the raw exhaust based on changes 

in exhaust flow. Although the CFR requires a ten point linearity verification on all flow 

measurements [2] this was not feasible on the DCS and make-up flow, which are both held 

constant at 2 lpm and cannot easily be varied. The steering committee elected to perform a spot 

check on the DCS flow to ensure its accuracy and verify the filter flow which includes both the 

total and make-up air flow. The filter flow is equal to the difference between the total flow and 

make-up air flow. Because the filter flow was designed to operate at a constant flow of 

approximately 28 lpm it was not practical or logical to verify the flow measurement over 10 even 
41

SwRI Project No. 14936.12

points down to zero flow as recommended in CFR Part 1065.307. After discussing the issue with 

Horiba and the steering committee it was decided to verify the filter flow over a range of plus 

and minus ten percent of its operating target, using eleven steps. The maximum range that the 

dilution flow could vary was approximately 21 to 31 lpm; the dilution flow was verified over this 

range in eleven even steps. The flow audits were initially performed with a TSI flowmeter, but 

this was replaced with a bubble flowmeter when it was discovered that the accuracy of the TSI 

flowmeter degraded as the flow deviated from atmospheric pressure. The results from the 

linearity verifications are shown in Table 12 for the Horiba PEMS.

TABLE 12. LINEARITY VERIFICATIONS FOR HORIBA PEMS

Horiba-1 narrowly missed the passing the linearity verification for total flow with a slope of 

0.97. This check was repeated several times without passing. However, the error as a percent of 

Verification Description Intercept Slope SEE R2

Measured 0.29 0.98 0.05 1.000

Linearity Criteria 0.31 0.98-1.02 0.62 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.07 0.97 0.03 1.000

Linearity Criteria 0.30 0.98-1.02 0.60 0.99

Pass / Fail Pass Fail Pass Pass

Measured 0.08 1.00 0.03 0.999

Linearity Criteria 0.31 0.98-1.02 0.61 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.18 0.98 0.03 1.000

Linearity Criteria 0.30 0.98-1.02 0.60 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.05 1.02 0.04 1.000

Linearity Criteria 0.31 0.98-1.02 0.62 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.01 0.99 0.04 0.999

Linearity Criteria 0.30 0.98-1.02 0.60 0.99

Pass / Fail Pass Pass Pass Pass

Dilution Flow

Filter Flow

Filter Flow

Horiba1

Horiba2

Horiba3

Dilution Flow

Filter Flow

Dilution Flow
42

SwRI Project No. 14936.12

point was better than 0.5 percent for all eleven points. This result is a problem with applying the 

linearity criteria from Part 1065 to a flow is not verified over the range from zero to full scale. 

Given the excellent agreement on a point-by-point basis, the steering committee elected to 

proceed without taking corrective action. 

The linearity verification results for the Sensors PEMS are shown in Table 13.
43

SwRI Project No. 14936.12

TABLE 13. LINEARITY VERIFICATIONS FOR SENSORS PEMS

Verification Description Intercept Slope SEE R2

Measured -0.01 1.00 0.00 1.000

Linearity Criteria 0.02 0.98-1.02 0.04 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.05 0.98 0.08 0.999

Linearity Criteria 0.09 0.98-1.02 0.17 0.99

Pass / Fail Pass Pass Pass Pass

Measured -0.04 1.01 0.01 1.000

Linearity Criteria 0.05 0.98-1.02 0.09 0.99

Pass / Fail Pass Pass Pass Pass

Verification Description Intercept Slope SEE R2

Measured 0.01 1.00 0.01 0.999

Linearity Criteria 0.01 0.98-1.02 0.03 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.07 1.01 0.05 0.999

Linearity Criteria 0.07 0.98-1.02 0.13 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.02 1.01 0.06 0.998

Linearity Criteria 0.04 0.98-1.02 0.08 0.99

Pass / Fail Pass Pass Pass Pass

Verification Description Intercept Slope SEE R2

Measured 0.01 1.00 0.01 0.999

Linearity Criteria 0.01 0.98-1.02 0.02 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.01 1.00 0.05 1.000

Linearity Criteria 0.07 0.98-1.02 0.14 0.99

Pass / Fail Pass Pass Pass Pass

Measured 0.01 0.96 0.01 1.000

Linearity Criteria 0.02 0.98-1.02 0.04 0.99

Pass / Fail Pass Fail Pass Pass

Major Dilution Flow

Sample Flow

Major Dilution Flow

Minor Dilution Flow

Minor Dilution Flow

Sensors2

Sensors1

Sensors3

Sample Flow

Major Dilution Flow

Minor Dilution Flow

Sample Flow
44

SwRI Project No. 14936.12

The Sensors PEMS had the capability of performing a self-audit using 1065 criteria. An external 

TSI flowmeter was provided as part of the Sensors equipment and its measurements were 

recorded by the Sensors software to determine perform linearity verifications on the dilution and 

sample flows. The total dilution flow is calculated by the addition of the major and minor 

dilution flows, so these two measurements are audited independently. The Sensors PEMS was 

able to pass all but the minor dilution flow for Sensors-3. This verification was performed 

repeatedly without passing. Since the total dilution flow would still pass a linearity verification 

in this case, no further action was taken. In the case of both the Horiba and the Sensors, the 

absolute accuracy of the flows are important in determining the mass of PM emitted. Because the 

AVL system measures the soot concentration in real time using a constant dilution ratio, only the 

accuracy of the dilution ratio matters when calculated the soot mass. For this reason the dilution 

ratio was audited instead of the total and dilution flow rates which are used to calculate the 

dilution ratio. It was also not possible to vary the total flow, which is held constant during 

normal operation. Although the dilution ratio was maintained at a constant of 5 throughout 

official testing, a six point check was performed ranging from 2 to 6 in steps of 1. Table 14

shows the results for the linearity verifications for the AVL units.

TABLE 14. LINEARITY VERIFIC ATIONS FOR AVL PEMS

The MSS dilution ratio was initially verified using both flow measurements as well as CO2

measurements. Because the AVL PEMS had an internal CO2 measurement it would have been 

possible to use a CO2 span bottle to verify the dilution ratio, unfortunately it was not possible to 

span the CO2 sensor (ie a dilution ratio of 1). The results shown in Table 14 were generated 

Verification Description Intercept Slope SEE R2

Measured 0.01 0.96 0.01 1.000

Linearity Criteria 0.07 0.98-1.02 0.15 0.99

Pass / Fail Pass Fail Pass Pass

Verification Description Intercept Slope SEE R2

Measured 0.03 0.97 0.02 1.000

Linearity Criteria 0.07 0.98-1.02 0.14 0.99

Pass / Fail Pass Fail Pass Pass

Verification Description Intercept Slope SEE R2

Measured 0.05 1.02 0.00 1.000

Linearity Criteria 0.07 0.98-1.02 0.14 0.99

Pass / Fail Pass Pass Pass Pass

AVL1

AVL2

AVL3

Flow Based Dilution Ratio

Flow Based Dilution Ratio

Flow Based Dilution Ratio
45

SwRI Project No. 14936.12

using TSI flowmeters to measure the total and dilution flow and calculate the dilution ratio in the 

same manner as the PEMS. AVL-1 and AVL-2 were both unable to pass the slope criteria but 

were within three percent of point across the six point verification. The steering committee 

agreed to accept the dilution ratio accuracy tolerance of three percent.

3.1.2 Exhaust Flow

Official linearity verifications were not conducted on the PEMS exhaust flow meters at SwRI. 

The steering committee decided that a calibration from the manufacturer was sufficient so long 

as the flow meter was within five percent of the lab during engine testing. The three flow meters 

tested from Horiba and the three from Sensors were all within found to be within five percent of 

the lab measurement. An unofficial linearity verification was performed using the data from the 

steady state engine testing. Each steady state measurement was considered a data point which 

resulted in between 64 and 84 data points per exhaust flow meter. The linearity plots are shown 

in Figure 15 for PEMS-1, Figure 16 for PEMS-2, and Figure 17 for PEMS-3.

FIGURE 15. LINEARITY CHECK ON PEMS-1 EXHAUST FLOW DURING STEADY 

STATE ENGINE TESTING

y = 0.977x - 16.188

R2 = 0.998

y = 0.995x - 15.427

R2 = 0.999

400

500

600

700

800

900

1000

1100

1200

400 500 600 700 800 900 1000 1100 1200

PEMS Exhaust Flow (m3/hr, 20°C)

Lab Exhaust Flow (m3/hr, 20°C)

Horiba-1 Sensors-1
46

SwRI Project No. 14936.12

FIGURE 16. LINEARITY CHECK ON PEMS-2 EXHAUST FLOW DURING STEADY 

STATE ENGINE TESTING

y = 0.995x - 17.619

R2 = 1.000

y = 1.021x - 37.608

R2 = 0.997

400

500

600

700

800

900

1000

1100

1200

400 500 600 700 800 900 1000 1100 1200

PEMS Exhaust Flow (m3/hr, 20°C)

Lab Exhaust Flow (m3/hr, 20°C)

Horiba-2 Sensors-2
47

SwRI Project No. 14936.12

FIGURE 17. LINEARITY CHECK ON PEMS-3 EXHAUST FLOW DURING STEADY 

STATE ENGINE TESTING

Each of the six exhaust flow meters was able to pass the standard error, slope, and correlation 

coefficient criteria for a raw exhaust flow measurement specific in CFR Part 1065.307 however 

only one was able to pass the intercept criteria (Sensors-3). This is likely due in part to the fact 

that the measurements did not extend below 470 m3/hr making it more difficult to pass an 

intercept criteria that assumes evenly spaced data points extended down to zero. Treating the 

exhaust flow measurement during engine testing as a linearity verification was for informational 

purposes only.

4 ENGINE DYNAMOMETER TESTING AND RESULTS

4.1 TESTING OBJECTIVE

The purpose of the engine dynamometer testing was to characterize the bias and precision errors 

of the PEMS during steady state and transient engine operation. During steady state engine 

operation the PEMS measurements were compared with that of the CVS filter-based PM 

measurement to characterize the bias in each of the three PEMS. The transient engine testing was 

used to determine the precision of each PEMS by quantifying the variability of the PEMS 

measurement over a series of repeated transient NTE events.

y = 0.986x - 13.106

R2 = 0.999

y = 0.987x + 3.218

R2 = 0.996

400

500

600

700

800

900

1000

1100

1200

400 500 600 700 800 900 1000 1100 1200

PEMS Exhaust Flow (m3/hr, 20°C)

Lab Exhaust Flow (m3/hr, 20°C)

Horiba-3 Sensors-3
48

SwRI Project No. 14936.12

4.2 EXPERIMENTAL SETUP

4.2.1 Engine and Sampling System

The test plan initially called for dynamometer testing to be performed on two different engines, 

but funding constraints reduced testing to a single engine. Initial test cell work was performed 

using a 6.4 liter light heavy-duty diesel engine provided by Navistar. The engine used to generate 

all official steady state and transient data was a 2007 Volvo MP7 provided by Volvo Powertrain. 

The engine had a displacement of 10.8 liters and was rated at 280 kilowatts (375 horsepower). 

The engine was equipped with a variable geometry turbocharger (VGT) and a water-cooled high 

pressure exhaust gas recirculation (EGR) loop. The engine intake system was connected to a 

test-cell water-cooled intercooler. The engine is shown in Figure 18.

FIGURE 18. VOLVO MP7 INSTALLED IN A CVS TEST CELL

The engine was also equipped with a close-coupled diesel oxidation catalyst (DOC) and a 

diesel particulate filter (DPF) combination. For the purpose of producing higher PM emission 

levels a bypass was created around the aftertreatment system to allow an adjustable amount of 
49

SwRI Project No. 14936.12

exhaust flow around the DPF. A DOC was added to the bypass so that all of the PM in the 

exhaust would pass through an oxidation catalyst simulating a scenario of a cracked DPF. A 

picture of the bypass is shown in Figure 19.

FIGURE 19. DPF BYPASS WITH DOC

The original stock aftertreatment was located in the main leg of the exhaust, while a separate 

catalyst was procured for the bypass leg. The DOC was 76 millimeters (six inches) in diameter 

with a length of 152 millimeters (12 inches). Three butterfly valves were placed in the exhaust 

system to control the amount of exhaust passing through each leg. The DPF was regenerated via 

an exhaust fuel injection system. For all testing the bypass was open to some degree, however, 

the bypass leg was closed when active regenerations were performed on the DPF. The DPF 

bypass went through multiple iterations until the proper PM level was achieved during steady 

state testing. A PM level of 0.025 g/hp-hr was easily obtainable during transient cycles, however 

it was extremely difficult to obtain this same PM level during steady state engine operation. 

Table 15 lists the five different configurations of the bypass that were tested.
50

SwRI Project No. 14936.12

TABLE 15. LIST OF DPF BYPASS CONFIGURATIONS

Iteration Pipe 

Diameter Inlet Probe DOC Diameter Outlet Probe

1 3 No 3 1", Upstream

2 3 No 3 1", Downstream

3 3 3 in, Upstream None 1", Downstream

4 3 3 in, Upstream 6 1", Downstream

5 (Final) 4 3 in, Upstream 6 3", Downstream

In the final configuration (iteration 4), two butterfly valves in the main leg of the exhaust were 

completely closed with only a one or two millimeter gap between the valve and the exhaust pipe. 

This not only forced a majority of the exhaust through the bypass, but significantly raised the 

exhaust backpressure. Based on measurements upstream and downstream of the bypass with the 

AVL PEMS it was estimated that well over 50 percent of the exhaust was routed through the 

bypass in the final configuration.

A diagram of the bypass system is shown in Figure 20. 

FIGURE 20. SCHEMATIC OF ENGINE DYNAMOMETER EXPERIMENTAL SETUP
51

SwRI Project No. 14936.12

The schematic is not to scale so that some distances may appear incorrectly. There are more than 

10 diameters between the mixing point of the bypass and the first PEMS sampling position for 

the Sensors Semtech PPMD. Additionally there is approximately 10 diameters in between each 

of the PEMS sampling locations so that any flow disturbances caused by a different pitot tube or 

sample probe should not affect the other PEMS. The portion of exhaust between the first and the 

last sampling location is insulated to minimize the cooling of the exhaust and thermophoretic 

deposition of particles. The last sampling probe is that of the Horiba OBS-TRPM which is just 

upstream of the entrance into the CVS tunnel. The gaseous PEMS units (Sensors Semtech DS 

and Horiba OBS-2200) were used only for data acquisition and trigger signals. No PEMS 

gaseous emissions were recorded during this program. The Semtech DS was used with both the 

Sensors PPMD and the AVL MSS while the Horiba OBS-2200 was used in conjunction with the 

Horiba OBS-TRPM.

The outlet of the aftertreatment was routed to a constant volume sampling (CVS) tunnel for 

emissions measurement. The CVS consists of a positive displacement pump and an upstream 

heat exchanger; it was maintained at a nominal flow rate of 3,740 m3/hr (2,200 scfm) for this 

testing. The dilution air is extracted through a filter pack from a temperature and humidity 

controlled area. The portion of the CVS tunnel that is exposed to the test cell environment 

upstream of the exhaust is insulated to prevent heating of the dilution air from the test cell. The 

particulate matter samples were extracted from the CVS as shown in Figure 20.

Intake air flow was measured by a laminar flow element (LFE) with a maximum flow of 1,700 

m3/hr (1,000 scfm). The LFE was oriented so that there were 10 diameters of straight pipe before 

the inlet and after the outlet to minimize flow disturbances. The fuel flow was measured by a 

Micro-Motion flow meter. The addition of intake air flow and fuel flow was used to determine 

the exhaust flow using the equations from CFR Part 89. The exhaust flow was used to calculate 

the exhaust PM concentration from the CVS filter as well as a check on the PEMS exhaust flow 

meters. A raw CO2 analyzer was used in combination with the exhaust flow measurement to 

calculate a raw carbon balance fuel flow which was compared with the measured fuel flow and 

the dilute carbon balance fuel flow as a quality check.

The PM sampling system consisted of a 47mm Whatman Teflo filter, a fine metal screen 

backing, and a plastic filter cartridge. The total flow is operated at a nominal flow of 57 slpm (2 

scfm) which results in a filter face velocity of approximately 100 cm/s. The nominal dilution 

flow is 28 slpm (1 scfm) resulting in a dilution ratio of 2 and a sample flow of 28 slpm (1 scfm). 

The system was heated to 47°C and had an approximate residence time of 0.8 seconds. A 

cyclone with a 2.5 micron cutpoint at 17 slpm was positioned just downstream of the sample 

probe.
52

SwRI Project No. 14936.12

4.2.2 Sensors PPMD

The PPMD was the PEMS unit installed closest to the outlet of the bypass, approximately 15 

diameters downstream. An experiment was conducted to ensure the flow was fully mixed at this 

location as will be discussed later. The PPMD was installed in the horizontal orientation as 

shown in Figure 21.

FIGURE 21. PPMD INSTALLED IN THE TEST CELL

The PPMD was equipped with a long straight pipe containing its exhaust flow meter (EFM) just 

upstream of the 90 degree elbow from which the sample is extracted. The dilution in the Sensors 

system takes place inside the instrument. The most common installation of the PPMD is in the 

vertical position, although it requires only the rotation of the moisture traps to properly operate 

the PPMD in the horizontal orientation. The Sensors PEMS was equipped with two stages of 

dilution known as MPS1 and MPS2. Although some preliminary testing was conducted using 

both stages of dilution, the steering committee decided to use only a single stage of dilution for 

all official tests. The PPMD is a proportional sampling system that varies its dilution ratio 

inversely with exhaust flow to maintain a minimum dilution ratio of 6 at the maximum exhaust 

flow rate of an engine. The PPMD measures PM using a Quartz Crystal Microbalance (QCM), 

which charges the particles using a corona needle, deposits the particle on a Quartz Crystal, and 

then measures the change in frequency of the crystal to determine the mass deposited. The 

PPMD is a batch sampling device meaning it does not report PM concentration in real-time but 
53

SwRI Project No. 14936.12

instead reports a single mass value per event. Because each crystal requires a pre and post 

frequency measurement to determine mass, a total of eight crystals are included to allow for 

continuous operation by switching crystals. Crystal sampling begins as soon as the engine enters 

the NTE zone and stops as soon as the engine exits the NTE zone. One of the of the eight 

crystals was used as a reference crystal to adjust the measurements for changes in temperature 

and pressure. This left up to seven crystals available for measurement although it was common to 

have one or two crystals not working on any given test. The timing of the samples during testing 

was designed around having a minimum of five working crystals available for measurement. The 

PPMD was included in the measurement allowance program because inertial microbalances had 

already been approved for PEMS applications in 40 CFR Part 1065.

4.2.3 AVL MSS

The AVL Micro Soot Sensor (MSS, also known as the Photo Acoustic Soot Sensor or PASS) 

was installed downstream of the PPMD in the middle of the vertical portion of the exhaust pipe 

leading to the CVS tunnel. The MSS is connected to the tunnel via a 3 meter heated sample line 

which was maintained at 150°C. The mixing of dilution air takes place in the dilution box just 

upstream of the sample probe so that the dilute sample is transported through the sample line. 

The MSS is shown in Figure 22.

FIGURE 22. THE AVL MSS
54

SwRI Project No. 14936.12

The MSS consists of two boxes shown in the above figure. The top box is the measuring unit 

which contains the resonance chamber for the soot measurement. The bottom box is the 

conditioning unit which contains the sample and dilution pumps as well flow controllers. The 

dilution pump is optional as the MSS can also provide dilution air via an external input of 300 

kPa of compressed air. The steering committee requested that the MSS operate using its internal 

dilution pump, since this is the way it would operate during in use testing. The MSS measures 

soot by heating the elemental carbon using a laser. When the soot is heated it emits a sound wave 

that is detected by a microphone. The MSS can report soot concentration on a 1Hz basis and uses 

a constant dilution ratio, which was set at 5 for all official testing. Because the MSS measures 

only soot and not total PM, it was included in this program as a partial participant. If both the 

Sensors and Horiba units could not complete the measurement allowance program it was to be 

considered for in-use. The AVL system does not have its own gaseous PEMS; instead it sends its 

concentration signal to the Sensors Semtech DS where the necessary gaseous emissions and 

J1939 broadcast information is recorded. For all testing in this program a single Semtech DS was 

used to record the signals from both the Sensors Semtech PPMD as well as the AVL MSS. The 

probe for the Horiba OBS-TRPM is located approximately 1.65 meters (5 feet) downstream of 

the MSS. The Horiba system contains two separate exhaust pieces that are each about 150 mm (6 

inches long). The upstream portion is a tail pipe adapter (TPA) which is a pitot tube exhaust flow 

measurement. The downstream pipe contained the probe for the PM sampling. The TPA and 

sampling probe can be seen in Figure 23.

FIGURE 23. THE PROBE AND TPA FOR THE HORIBA OBS-TRPM
55

SwRI Project No. 14936.12

4.2.4 Horiba TRPM

The Horiba system is a proportional sampling device that varies its dilution ratio in the same 

manner as the PPMD. The dilution air is introduced just downstream of the probe before the 

heated sample line. The point of dilution can be seen in Figure 23 where the three stainless steel 

lines converge into the stainless steel cylinder. The OBS-TRPM uses a TSI EAD (referred to 

here as a DCS) real time particle instrument to measure the particle concentration on a second by 

second basis and collects PM on a gravimetric filter simultaneously. The filter weight gain is 

used to provide a calibration constant to the real time particle signal and apportion the PM mass 

appropriately. The DCS instrument measures continuously, but the filter is designed to sample 

during valid NTE event operation. The filter sampling begins after five seconds in the NTE zone 

and will continue for a minimum sample time of 30 seconds even if the engine is no longer in the 

NTE zone. Because the EPA’s PM standard is based on gravimetric filter analysis, the Horiba 

system was included in the program. The OBS-TRPM is comprised of several different boxes 

including the heated enclosure (HE), the diffusion charge sensors (DCS), the electrical enclosure 

(EE), and the mechanical enclosure (ME). The HE contains the 47mm filter holder, and the DCS 

is the real time particle sensor. Dilution air was provided using a commercially available oil-less 

compressor. The Horiba OBS-2200 gaseous PEMS was used to log the ECM J1939 broadcast, 

measure the exhaust flow, and provide an NTE trigger to the TRPM to start filter sampling. The 

combined TRPM-2200 system contains a total of 6 boxes plus an external compressor. The 

components of the Horiba system are depicted in Figure 24.
56

SwRI Project No. 14936.12

FIGURE 24. THE HORIBA OBS-TRPM AND OBS-2200

The TRPM used the same 47mm Whatman teflo filter, metal screen, and plastic cartridge as the 

CVS system. All weighing and conditioning of both the CVS and the TRPM filters was 

conducted in the SwRI filter weighing room. The filter weighing room is maintained at a 

temperature of 22 ± 1°C with a dewpoint of 9.5 ± 1°C in accordance with CFR Part 1065.190. 

Filters were stabilized in the weighing environment for at least 1 hour prior to both the initial and 

final weights. Each filter was weight was determined by the average of three weights on a scale 

with a resolution of 0.1 μg. 

4.3 BYPASS MIXING VERIFICATION

The flow from the DPF bypass was reintroduced into the main exhaust stream using a 76 mm (3 

inch) probe facing downstream with an orifice near the tip of the probe to promote mixing. 

Testing was conducted to ensure the exhaust flow was fully mixed prior to the first sampling 

location, which was occupied by the Sensors PEMS. Two probe orientations were created at the 

spot where the Sensors sample was extracted. One of the probe orientations was parallel to the 

upstream exhaust elbow and one was perpendicular to the elbow. The orientation of the AVL 

sample probe is shown in Figure 25.
57

SwRI Project No. 14936.12

FIGURE 25. EXPERIMENTAL SETUP FOR MIXING VERIFICATION

At each of the probe orientations the MSS was used to measure the exhaust soot concentration 

using a variable length probe that could traverse the length of the 127 mm (5 inch) exhaust pipe. 

There were five sample locations for each orientation as shown in Figure 26.

FIGURE 26. MIXING VERIFICATION SAMPLE LOCATIONS
58

SwRI Project No. 14936.12

Each of the positions were one inch apart, with position 3,8 being at the center of the exhaust 

pipe. For a measurement, the probe was started at location 1, moving down to 5 and then back to 

1. The probe was then moved to position 6 moving down to position 10 and back to position 6. 

The MSS measurement was recorded for 80 seconds at each position, with the average of the last 

30 seconds used for comparison. The steady state modes with the highest and lowest exhaust 

flow rates were chosen to perform the mixing verification, to ensure proper mixing over the 

entire range of test conditions. 

The mixing verification was performed several times with different bypass configurations with 

similar results in all cases. Only the results from the final mixing verification are presented since 

the other results from different configurations are not relevant to the data in this report. The data 

from the final verification is presented in Figure 27.

FIGURE 27. MIXING VERIFICATION RESULTS 

The data is presented as the ratio of the average concentration at each location to the centerline 

concentration. The data was normalized in this way not only to remove some of the engine 

variability but also so that the two exhaust flow conditions with different soot concentrations 

could be compared on the same plot. Each data point represents the average of two samples 

taken at each location. The highest deviation from the centerline concentration was four percent 

at location 10, low flow which was considered acceptable, especially given that no trends were 

observed in the data. A paired t-test using equal variance was also performed on the data relative 

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1 2 3 4 5 6 7 8 9 10

Concentration/Centerline Concentration

High Flow Low Flow
59

SwRI Project No. 14936.12

to the centerline position for the parallel orientation using the high and low flow. Locations 6 

through 10 passed the t-test, except for location 10 at low flow, although the difference in the 

mean value between locations 10 and 7 was less than 4 percent. The steering committee was 

satisfied with the mixing results and directed SwRI to move forward with the program without 

any further modifications for the bypass setup. 

4.4 PEMS LOSS CORRECTIONS

Each PEMS manufacturer was given the opportunity to correct their final PM measurement to 

account for various particle losses encountered during the sampling process. Any loss correction 

had to be presented to the steering committee for approval before it was implemented in the 

program. Sensors and AVL both chose to implement loss corrections, while Horiba declined 

stating that the losses in their system were already comparable to the losses in a CVS system.

4.4.1 Sensors PPMD Loss Correction

Sensors conducted work under a separate project at SwRI to experimentally assess the losses in 

the PPMD. David Booker presented the results of this work along with the proposed Sensors loss 

corrections at the meeting on December 10th

4.4.2 AVL MSS Loss Correction

, 2008 at SwRI. The final PPMD loss correction 

included thermophoretic, electrostatic, and CVS loss factors. The CVS loss correction factor was 

meant to estimate the particle losses in the CVS system, since this is the standard to which the 

PPMD is compared. Although typical loss correction factors will increase the estimated PM 

concentration, the CVS correction factor actually decreased the PPMD estimated concentration. 

A total CVS loss of 15 percent was estimated by Sensors based on general experience rather than 

specific data. The total loss correction was estimated to increase the PM concentration by 5 to 10 

percent when including the electrostatic and thermophoretic loss factors. The proposed loss 

corrections were accepted by the steering committee and implemented in the Sensors PPMD post 

processor. All Sensors data in this report includes these correction factors unless otherwise 

stated.

The proposed AVL loss correction was presented at the meeting on November 12th

M mss t t q (t) ( L(T (t)))dt soot = ∫ + ∆ ex 1+ ex ( )

, 2008 at 

SwRI. The loss correction implemented by AVL was intended to correct for the thermophoretic 

losses in the system and is based off a paper by Stratmann et al [6]. The equation for the loss 

correction is shown below:

( ) 











+

< = 150 300 otherwise0 if T 150

a b(T- )/

L T
60

SwRI Project No. 14936.12

The magnitude of the correction is temperature dependent and was estimated to be 

approximately 10 percent in most cases. This correction was accepted by the steering committee 

and implemented in the AVL Concerto post processor. 

At the meeting at SwRI on December 10th

4.4.3 AVL MSS Total PM Correction

, 2008, AVL stated that their loss correction was 

currently capped at a maximum loss of 25 percent regardless of the calculated value. AVL 

requested approval to remove this limitation and allow the equations output to be the correction, 

regardless of its magnitude. This change was accepted by the steering committee. 

The AVL Concerto post processor includes a function that estimates the total PM based on the 

measured soot, exhaust temperature, and total hydrocarbon concentration, along with a number 

of adjustable input parameters including the volume of the catalyst, the light off temperature of 

the catalyst and the sulfur content of the fuel. Although this calculation was not approved for use 

with the official measurement allowance data, a portion of the AVL data was processed to 

examine the results from the total PM model. All AVL data presented in this report refers to the 

soot concentration corrected for losses unless otherwise stated.

4.5 STEADY STATE TESTING PROCEDURE

Originally the test plan called for two different engines at three different emission levels of PM: 

DPF out, 0.02 g/hp-hr level, and a 0.03 g/hp-hr level. The DPF out level is simply whatever the 

emissions happen to be with no bypass which was well below the 2007 standard of 0.01 g/hp-hr. 

The 0.02 and 0.03 g/hp-hr levels would be set by adjusting the DPF bypass to produce the 

corresponding brake-specific PM number from the CVS filter. Due to funding limitations the 

testing was reduced to a single engine at a single emission level. Because 0.03 g/hp-hr will be 

used for the first year of compliance testing and 0.02 g/hp-hr is used with the following years, it 

was important to investigate the performance of the PEMS covering these levels. For this reason 

an average of the two threshold PM levels, 0.025 g/hp-hr, was used as the target.

The objective of the steady state testing was to evaluate the bias and precision of the PEMS 

using 180 data points for each PEMS manufacturer. The 30 points consists of six steady state 

modes of engine operation (6), 10 repeats (10), one emission level (1), one engine (1), and three 

different PEMS units (3), 6*10*1*1*3=180. A PM steady state error surface, ( ) g ΔSS PM m mol , was 

developed for each PEMS manufacturer so that there are three steady state PM error surfaces for 

use with calculation methods 1 and 2. For calculation method 3, the AVL MSS will have a 

unique ΔSS PM m based on the calculations for method 3. As mentioned previously, the Sensors 

and Horiba PEMS will only use methods 1 and 2. To pick six steady engine modes for steady 

state testing screening tests were performed using the 80 points Cummins cycle and measuring 

the PM levels with the AVL MSS and the TSI Engine Exhaust Particle Sizer (EEPS). The EEPS
61

SwRI Project No. 14936.12

primarily provides size distribution information, but mass concentration can be inferred using an 

assumed density. Since the AVL is the only of the three instruments that can report a real time 

mass concentration without further efforts such as filter weighing and post processing it was 

chosen to perform the screening work. The Cummins cycle steps through 80 steady state engine 

modes as a transient cycle. The engine is stepped through 10 different speeds and eight engine 

loads at each of the selected speeds. The minimum speed of the cycle is the minimum NTE 

speed, and the minimum torque is 30 percent of the torque at the given speed meaning that the 

cycle effectively maps the NTE zone. Figure 28 shows the speed and torque points of the cycle.

FIGURE 28. SPEED AND LOAD FOR THE 80 POINTS CYCLE

The engine remained at each of the points for 88 seconds with a one second transition between 

modes. Because the speed and load were only incrementally changed in between each mode, it 

was assumed that each point would stabilize relatively quickly. The fact that this cycle was used 

only as a method of screening also contributed to the decision to change modes quickly. Even 

with such short modes, the total cycle length was 2 hours. To allow for purging, and calibrating 

the gas analyzers, the cycle was divided into two 40 points cycles each lasting one hour. The gas 

analyzers were necessary because this cycle was also used to tabulate ECM fuel rate errors for 

this error surface. 

The PM emissions were estimated by measuring each of the 80 modes with the EEPS and MSS, 

then choosing two of the modes to perform a filter measurement and compare the ratio of the 

filter measurement to the EEPS and MSS as an estimate of the CVS filter BSPM at each of the 

0

20

40

60

80

100

120

0 20 40 60 80 100 120

Engine Torque (%)

Engine Speed (%)
62

SwRI Project No. 14936.12

80 modes. Although this method is not highly accurate, it provided a way to quickly obtain rough 

estimates of the engine PM levels over a wide range of speed and torque conditions. Of the initial 

80 points, only 12 were estimated to produce brake-specific PM of greater than 0.02 g/hp-hr. The 

final six modes were chosen with the intent of covering as much of the NTE zone as possible 

while still maintaining high BSPM levels, and a range of PM exhaust concentrations. Several 

iterations of adjusting the bypass, taking filter measurements, and narrowing down the number of 

points occurred before the steering committee approved the final six points for steady state 

testing. Originally the DPF bypass was adjusted to produce the 0.025 g/hp-hr of PM based on a 

filter measurement during a short version of the NTE transient cycle. The PM emissions at 

steady state were much lower at the same bypass setting. Since it was desirable to conduct the 

steady state testing at the same PM levels as the transient testing, it was necessary to adjust the 

system to allow a much greater amount of the flow through the bypass. Six points were chosen 

out of the twelve to provide a range of PM concentration, exhaust flow rates, and engine 

operating conditions. The six points that were chosen are shown in Figure 29 along with the NTE 

zone.
63

SwRI Project No. 14936.12

FIGURE 29. FINAL SIX STEADY STATE MODES

Although the MSS was used to screen the 80 points, the actual concentration at each position 

was verified with CVS filter measurements before selecting the points. The exhaust PM 

concentration was calculated by multiplying the CVS PM concentration by the CVS dilution 

ratio. The CVS dilution ratio was calculated by dividing the average CVS flow rate by the 

average exhaust flow rate. The CVS dilution ratio ranged from 3 to 7.5 resulting in an overall 

dilution ratio of 6 to 15 when including the secondary PM filter dilution. 

The steady state testing was conducted as a modal transient cycle with each mode repeated twice 

for a total of 12 modes per cycle. 6 different cycles were created with the order of the modes 

randomized in each cycle. Table 3 lists the sample order of the modes in the 6 steady state 

cycles.

0

500

1000

1500

2000

2500

500 1000 1500 2000 2500

Engine Torque (Nm)

Engine Speed (rpm)

Lug Curve NTE S Line NTE T-S Line NTE P-S Line SS Points

Mode 1

Mode 2

Mode 3

Mode 4

Mode 5 Mode 6
64

SwRI Project No. 14936.12

TABLE 16. SAMPLE ORDER FOR STEADY STATE CYCLE TESTING

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6

Sample 1 3 1 3 3 4 6

Sample 2 6 4 1 4 2 1

Sample 3 5 3 6 1 1 4

Sample 4 2 6 2 2 5 3

Sample 5 4 5 5 6 3 5

Sample 6 1 2 4 5 6 2

Sample 7 5 2 4 6 5 4

Sample 8 2 1 2 4 3 1

Sample 9 1 4 3 1 4 6

Sample 10 3 5 1 2 6 3

Sample 11 6 6 5 5 2 5

Sample 12 4 3 6 3 1 2

The steady state testing was conducted as a ramped modal cycle with the engine remaining at 

each operating condition for three minutes before the start of sampling. An external trigger from 

the lab was provided to each of the PEMS and the CVS filter system so that sampling would 

begin simultaneously for all instruments. The engine then remained at the operating condition for 

5 seconds after the end of sampling to ensure that no delay in the end of sampling by any of the 

PEMS caused part of the transition period to be captured as a sample. The engine remained at the 

condition for five seconds after sampling had finished to ensure all systems had finished 

sampling before the operating condition changed. The order of the modes was randomized and 

each mode was repeated twice within a single cycle for a total of 12 data points for each cycle. 
65

SwRI Project No. 14936.12

Six different cycles were created, which would create a total of 72 data points. Although the 

target was only 60 valid points for each set of PEMS, in practice several cycles had to be 

repeated to collect enough valid data. An example of one of the steady state cycles is shown in 

Figure 30.

FIGURE 30. EXAMPLE OF STEADY STATE CYCLE

To ensure an accurate filter weight, a filter weight gain of 100 μg was targeted for each mode. 

By ensuring that this amount of material is collected on the filter for each test, the weighing 

variability and tunnel background contribution could be minimized. To produce a similar filter 

weight gain at six different steady state modes with different mass rates, the sampling time was 

adjusted for each mode to meet this target. The sample time ranged from 50 seconds to 245 

seconds. Since the sampling time for each mode was different, the total length of time spent at 

each mode was different as well. The sample time for each mode for each round of the PEMS is 

shown in Figure 31.

0

500

1000

1500

2000

2500

0 500 1000 1500 2000 2500 3000 3500 4000

Torque (Nm), Engine Speed (rpm)

Time (sec)

Speed Torque
66

SwRI Project No. 14936.12

FIGURE 31. STEADY STATE SAMPLE TIMES

Figure 32 shows the average filter weight gain for all six modes for each of the three PEMS.

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

PEMS 1 245 70 125 190 160 100

PEMS 2 240 50 100 175 90 60

PEMS 3 240 50 100 175 90 60

0

50

100

150

200

250

Sample Time (sec)
67

SwRI Project No. 14936.12

FIGURE 32. CVS FILTER WEIGHT GAIN FOR STEADY STATE TESTING

Before presenting any of the PM concentration data it should be mentioned the choice of units in 

this work. The test plan called for the PM concentration to be calculated in terms of μg/mol. 

Although it is an uncommon unit for describing mass concentration it was considered for

fundamental to use mol for volume rather than m3 in which case a standard reference condition 

must be defined. For reference, 1 mg/m3 is equal to 24.55 μg/mol. In several cases the values in 

mg/m3 are provided in parenthesis for reference, but all official data was calculated and plotted 

using μg/mol.

The preliminary steady state results from PEMS 1 were presented at the December 11th

Figure 33

, 2008 

meeting at SwRI. After reviewing the first set of steady state data, the steering committee felt 

that the concentrations from the six steady state points were not effectively covering the desired 

range. Five of the modes are clustered between 115 and 161 μg/mol (4.8 and 6.7 mg/m3) with the 

remaining mode at 325 μg/mol (13.5 mg/m3). At the recommendation of the steering committee 

the bypass setting was slightly adjusted for PEMS 2 and PEMS 3 in an attempt to fill in some of 

the region between 161 and 325 μg/mol. Due to a shift in the engine out PM emissions, it was 

possible to increase the concentration for modes 2, 3, 5, and 6 while maintaining the same levels 

for modes 1 and 4. In fact, the dampers were actually adjusted to flow less exhaust through the 

bypass indicating that the engine out PM had not only changed relatively between operating 

conditions but increased overall. shows the median CVS filter PM concentration for 

each of the three sets of PEMS.

0

20

40

60

80

100

120

140

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

CVS Filter Weight Gain (μg)

PEMS 1 PEMS 2 PEMS 3
68

SwRI Project No. 14936.12

FIGURE 33. STEADY STATE EXHAUST PM CONCENTRATION (μG/MOL)

The mode with the highest concentration, mode 2, increased up to nearly 423 μg/mol while the 

lowest concentration increased from 115 μg/mol on mode 5 to 132 μg/mol on mode 4. It is 

interesting to note that mode 5 and 6 increased in concentration by 87 and 64 percent, 

respectively while mode 4 increased by less than 2 percent. This increase in concentration came 

while opening the valve in the DPF leg of the exhaust, thereby increasing the flow of exhaust 

through the DPF. Modes 2 and 3 both shifted downwards between PEMS 2 and PEMS 3 without 

any change in the exhaust valve positions. Figure 34 shows the brake-specific PM values as 

measured by the lab reference for all three sets of PEMS.

0

50

100

150

200

250

300

350

400

450

500

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

PM Concentration, CVS Filter (μg/mol)

PEMS1 PEMS2 PEMS3
69

SwRI Project No. 14936.12

FIGURE 34. STEADY STATE BRAKE-SPECIFIC PM, CVS FILTER (MG/HP-HR) 

The brake-specific PM ranged from 15.7 mg/hp-hr to 43.5 mg/hp-hr.

4.6 DATA YIELD DURING STEADY STATE TESTING

The test plan called for a minimum of 10 valid data points for each mode for each PEMS, 

allowing for a minimum data set of 30 for each 5th and 95th

0

5

10

15

20

25

30

35

40

45

50

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

Brake-Specific PM, CVS Filter (mg/hp-hr)

PEMS1 PEMS2 PEMS3

percentile delta that was generated. In 

reality the target of 10 data points per PEMS per mode was not met in all cases due to additional 

points that were invalidated by post processing software that had been updated after the testing 

had been completed. While very few data points were removed during post processing for the 

Horiba and AVL PEMS, Sensors supplied SwRI with several new post processors after the 

completion of testing that invalidated a significant portion of the Sensors data. Since the 

information on what criteria would invalidate the data was not available at the time of testing, it 

was not possible to know how many additional tests would be required to achieve the necessary 

number of data points. The Sensor’s post processor was revised to include some points that were 

deemed valid data but had been excluded by the post processor. The final set of data for the 

Sensors PEMS had between 28 and 34 points per mode.
70

SwRI Project No. 14936.12

FIGURE 35. NUMBER OF VALID DATA POINTS FOR STEADY STATE TESTING

The total number of possible data points was either 40 or 41 depending on the mode. Samples 

were taken 41 times at each mode, but modes 2, 3, 4, and 6 each had one point excluded due to 

mishandling of the CVS filter. 

4.6.1 Data Yield During Steady State Testing

Table 17 shows the steady-state data yield by each of the PM-PEMS relative to the possible data 

yield obtained by the CVS. A total of 29 steady state cycles were conducted for all three PEMS 

with each of the six steady state modes repeated twice in each cycle for a total of 12 data points 

per cycle. 21 of these cycles were considered valid tests from the perspective of the function of 

the cycle command, NTE external trigger, filter sampling, and at least one or more of the PEMS 

capturing valid data. During these valid tests four data points were missed due to a mishandling 

of the CVS filter, but the rest were considered valid data points from the perspective of the lab 

measurements. Data from the PEMS was removed for a variety of reasons including problems 

with the datalogging, sampling, exhaust flow measurement, mechanical failures, and filter 

handling. Appendix A contains a complete list of the reasons data was excluded, but several 

notable problems will be discussed here. 

0

5

10

15

20

25

30

35

40

45

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

Number of Valid Data Points

Horiba Sensors AVL
71

SwRI Project No. 14936.12

TABLE 17. DATA YIELD BY EACH PM-PEMS

Possible Horiba Sensors 

- NewPP

Sensors –

Revised PP AVL

Mode 1 41 37 22 34 40

Mode 2 41 36 25 29 33

Mode 3 41 36 25 32 33

Mode 4 41 35 24 28 35

Mode 5 41 37 17 31 37

Mode 6 41 37 23 30 34

Total 246 218 136 184 212

The Horiba system requires an external source of compressed air capable of supplying 

approximately 30 lpm at 400 kPa or higher. A commercially available oil-less compressor was 

provided by Horiba for use with the system. Unfortunately this compressor had a tendency to 

stop working on quite a few occasions during testing. Three different compressors of the same 

model were provided and each experienced this problem. It was believed that the compressor 

was overheating and shutting off to protect itself, although changes in the test cell temperature 

did not seem to influence its performance. The compressor would begin to work again after 10 to 

15 minutes presumably once it had cooled off. If the compressor stopped working while the 

Horiba system was in operation, the system lost all of its dilution air once the small air tank had 

been depleted. This resulted in an undiluted exhaust stream being sampled onto the filter which 

would quickly overload the filter at the PM concentration levels used in this work not to mention 

fail proportionality required of the Horiba system. If the compressor stopped working at any 

point during an official test, the Horiba data for that test was voided. Figure 36 shows an 

example of a steady state cycle where the compressor stopped working.
72

SwRI Project No. 14936.12

FIGURE 36. HORIBA SAMPLE FLOW TO THE FILTER AND DILUTION FLOW 

WHILE COMPRESSOR STOPS

A problem occurred with Sensors 1 involving the auto zero function of its exhaust flow meter. 

Every hour the Sensors system would attempt to zero the exhaust flow meter by switching the 

pressure transducers to ambient for a period of time less than a minute. On the Sensors 1 the 

solenoid switching the pressure transducers from exhaust measurement to zero was not working 

properly causing the zero function to occur while pressures were being measured from the 

exhaust. Because the steady state cycle lasted longer than one hour, this would cause an 

erroneous exhaust flow measurement on the last two modes of each steady state cycle. Figure 37

shows an example of the EFM zero problem during a test.

-5

0

5

10

15

20

25

30

35

0 1000 2000 3000 4000

Flow (slpm)

Time (sec)

Sample Flow to Filter Dilution Flow
73

SwRI Project No. 14936.12

FIGURE 37. SENSORS EFM ZERO DURING STEADY STATE CYCLE

The EFM auto zero function was disabled for the remaining tests on Sensors 1, and for Sensors 2 

and 3.

4.7 ACCOUNTING FOR CVS VARIABILITY DURING STEADY STATE TESTING

The steady state data has variability due to the PEMS, CVS, and test article. By computing a 

paired difference for each data point, the test article variability is removed from the data. This 

concentration delta still contains variability associated with the PEMS and the CVS. The data 

collected in this program does not allow for independent assessment of the PEMS and CVS 

variability, but by assuming a CVS variability it becomes possible to assign the remaining 

variability to the PEMS. 

The following procedure for estimating the PEMS and CVS variability was proposed by Bill 

Martin at the December 12th 2008 meeting at SwRI and was accepted by the steering committee.

1. Since test article variation is the same for each individual observation by the CVS (xPEMS,i) 

and PEMS (xCVS,i

( ) i PEMS,i CVS ,i ∆ = x − x

), compute the paired differences,

.

0

200

400

600

800

1000

1200

1400

1600

0 1000 2000 3000 4000

Exhaust Flow (kg/hr)

Time (sec)

Sensors Lab

EFM Zero
74

SwRI Project No. 14936.12

These paired differences (i.e., delta values or concentration deltas) contain random variation 

from the CVS, random variation from the PEMS, and a mean offset between CVS and PEMS 

(bias error). 

2. Divide the entire set of delta values into j = 1 to M subsets based on the values of χCVSi. The 

data sets are not subdivided by engine operating mode or PEMS serial number, but only the 

level of the reference concentration.

3. For each subset, there are i = 1 to Nj

∆50, j

values. The median and the MAD are used as the 

descriptive statistics.

4. Calculate the median delta value, , the median absolute deviation of the ∆i values, 

MADj , and the estimate of the standard deviation of the CVS random error, CVSrandom j SD , . 

N j i j median i 1, 50, ( ) = ∆ = ∆

N j i

MADj median ij j 1, 50, ( ) == ∆ − ∆

∑

∑ =

=

= ⋅

j

j

N

i

CVS i

j

N

i

CVS filter i

j

CVSrandom j x

N

L

N

g SD

1

,

1

, ,

,

1

1

5μ

⋅ ∑

∑

= =

=

j

j

N

i

N CVS i

i

CVS filter i

CVSrandom j x

L

g SD

1 ,

1 , ,

,

5μ

CVS j

CVS filter j

CVSrandom j x

L

g SD ,

, ,

,

5 = ⋅ μ

where LCVS , filter,i is the PM sample filter loading. 5 μg is the assumed CVS variability as 

proposed by the steering committee based on a nominal filter loading of 100 μg.
75

SwRI Project No. 14936.12

For each subset j, calculate a corrected delta for each ∆i value, in subset j, as follows:

ij j

j

j

j CVSrandom j

ij ij j

j CVSrandom j

MAD

MAD SD

MAD SD

50

2 50,

2

,

2

50,

2

,

2

else

( 0.45495 ) ( )

then 

if 0.45495

∆′ = ∆

+ ∆ − ⋅ ∆′ = ∆ − ∆ ⋅

> ⋅

Note that this approach correctly passes through any significant offsets observed in the data. 

These offsets should be passed through even if they persist only for a subset of data, such as 

a given mode. 

5. The entire set of corrected delta values is then to be used to establish the error surface for the 

steady-state data. The 5th, 50th, and 95th delta values are used to establish the 1st, 50th, and 

99th

4.8 STEADY STATE TESTING RESULTS

percentile values which are the inputs to the Monte Carlo model. 

All steady state data presented has already been corrected for the steady state variability as 

mentioned above. The steady state concentration deltas from PEMS 1 for Horiba, Sensors, 

and AVL, respectively are shown in Figure 38, Figure 39, and Figure 40.
76

SwRI Project No. 14936.12

FIGURE 38. HORIBA-1 STEADY STATE PM CONCENTRATION DELTAS

FIGURE 39. SENSORS-1 STEADY STATE PM CONCENTRATION DELTAS

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Horiba 50th % Horiba 95th % Horiba 5th %

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Sensors 50th % Sensors 95th % Sensors 5th %
77

SwRI Project No. 14936.12

FIGURE 40. AVL PM CONCENTRATION DELTAS FOR STEADY STATE TESTING 

ON PEMS 1

Each point on the x-axis represents the median exhaust PM concentration from the CVS filter for 

a single mode. The y-axis represents the 5th, 50th, and 95th

Figure 41

percentile of the deltas, PEMS – Lab. 

As mentioned previously there was a clear gap in the data between 162 μg/mol and 325 μg/mol 

which represents a large portion of the target concentration range for a 0.025 g/hp-hr level. 

, Figure 42, and Figure 43 show the deltas for PEMS 2 where three of the modes were 

between 208 and 267 μg/mol. 

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

AVL 50th % AVL 95th % AVL 5th %
78

SwRI Project No. 14936.12

FIGURE 41. HORIBA-2 STEADY STATE PM CONCENTRATION DELTAS

FIGURE 42. SENSORS-2 STEADY STATE PM CONCENTRATION DELTAS 

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Horiba 50th Horiba 95th Horiba 5th

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 100 200 300 400

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Sensors 50th % Sensors 95th % Sensors 5th %
79

SwRI Project No. 14936.12

FIGURE 43. AVL-2 STEADY STATE PM CONCENTRATION DELTAS

Horiba-1 and Horiba-2 performed similarly with mode 2 showing a significant negative bias, 

mode 4 showing a positive bias and the other four modes closer to zero. Sensors-2 showed a 

much greater negative bias than Sensors-1. Sensors-1 had a 50th percentile of between 0 and -26 

μg/mol for five of the six modes while Sensors-2 was between -32 and -104 μg/mol for the 50th

percentile for the same five modes. In addition, mode 2 had a 5th percentile of -340 μg/mol at a 

reference concentration of 442 μg/mol. For the same mode on Sensors-1 the 5th percentile was -

157 μg/mol at a reference concentration of 326 μg/mol. AVL-2 was lower than AVL-1 with 50th

percentiles between -34 μg/mol and -70 μg/mol. The 50th

Figure 44

percentiles for AVL-1 were between 

13 μg/mol and -32 μg/mol. No indication of the changes in performance for these PEMS were 

discovered through the recommended checks and audits. The PM deltas for Horiba-3, Sensors-3, 

and AVL-3 are shown in , Figure 45, and Figure 46 respectively. 

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

AVL 50th % AVL 95th % AVL 5th %
80

SwRI Project No. 14936.12

FIGURE 44. HORIBA-3 STEADY STATE PM CONCENTRATION DELTAS

FIGURE 45. SENSORS-3 STEADY STATE PM CONCENTRATION DELTAS

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Horiba 50th Horiba 95th Horiba 5th

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Sensors 50th % Sensors 95th % Sensors 5th %
81

SwRI Project No. 14936.12

FIGURE 46. AVL-3 STEADY STATE PM CONCENTRATION DELTAS

Horiba-3 and Sensors-3 are within the same ranges of their first two instruments, while AVL-3 

produced an extremely high bias at the highest concentration (mode 2). The 50th

When the data from all three PEMS is considered, the range of concentrations from 5 mg/m3 to 

18 mg/m3 is covered, although there is still a majority of the data located between 5 and 7 mg/m3 

as a result of the data from PEMS 1. The individual deltas for the Horiba, Sensors, and AVL are 

shown in 

percentile of 

mode 2 for AVL-3 was 84 μg/mol, while it was 13 μg/mol and -62 μg/mol for AVL-1 and AVL2, respectively. 

Figure 47, Figure 48, and Figure 49, respectively. Please note that each point represents 

a single measurement, not pooled data.

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

AVL 50th % AVL 95th % AVL 5th %
82

SwRI Project No. 14936.12

FIGURE 47. STEADY STATE CONCENTRATION DELTAS FOR HORIBA

FIGURE 48. STEADY STATE CONCENTRATION DELTAS FOR SENSORS

-400

-300

-200

-100

0

100

200

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Lab PM Concentration Reference (μg/mol)

Horiba-1 Horiba-2 Horiba-3

-400

-300

-200

-100

0

100

200

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Lab PM Concentration Reference (μg/mol)

Sensors-1 Sensors-2 Sensors-3
83

SwRI Project No. 14936.12

FIGURE 49. STEADY STATE CONCENTRATION DELTAS FOR AVL

Because the reference concentration of several of the modes were similar, the steering committee 

decided to group the data by reference concentration rather than by operating mode. The data 

was split up into groups of approximately 20 data points based on reference concentration for 

developing the steady state PM error surface. The Horiba data was grouped into 11 sets, the 

Sensors data into 9 sets, and the AVL data into 10 sets. 

Figure 50, Figure 51, and Figure 52 show PEMS PM concentration plotted against the PM 

concentration determined by the reference laboratory filter method for steady-state NTEs. These 

plots mainly show the qualitative PEMS to PEMS scatter relative to the filter method. 

Figure 53 shows a linear regression between each set of PM-PEMS and the CVS filter method. 

Good correlation was observed between the MSS and lab, with as coefficient of determination 

(R2

-400

-300

-200

-100

0

100

200

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Lab PM Concentration Reference (μg/mol)

AVL-1 AVL-2 AVL-3

) of 0.84 and a slope of 0.89. The slope suggests that the MSS PM concentration is 11 percent 

lower than that determined by the lab. This trend is expected since the MSS is measuring soot 

and the lab reports total PM. The slope seemed to be high because the MSS PM concentration in 

the range between 15 and 20 mg/m3 was higher than that of the lab. 
84

SwRI Project No. 14936.12

The linear regression between the Horiba TRPM and the lab resulted in R2 of 0.55 and a slope of 

0.86, indicating some correlation. After further investigation, it was recognized that the weak 

correlation was due to Mode 4 (high speed, light load) of the SS testing, with concentration 

levels of about 5 mg/m3. By removing this mode from the data, a R2 of 0.85 was obtained and the 

slope was moved from 0.86 to 0.88. It is likely that Mode 4, could have resulted in 

overestimation of PM due to nanoparticle formation with the Horiba dilution system, although 

CVS testing at this condition did not show a nanoparticle mode. Based on the slope of the 

correlation, the Horiba PM concentration was 14 percent lower than that reported by the lab. 

The linear regression between the Sensors PPMD and the lab results in R2 of 0.34 and a slope of 

0.64. The weak correlation was due to data scatter. Except in the narrow range between 5 and 7 

mg/m3, the Sensors PPMD showed underestimated PM. Based on the slope, the Sensors PPMD 

PM concentration was 36 percent lower than that reported by the lab. 

FIGURE 50. STEADY STATE HORIBA PEMS PM CONCENTRATION VERSUS THE 

LABORATORY REFERENCE

0

50

100

150

200

250

300

350

400

450

500

0 100 200 300 400 500

PEMS PM Concentration (μg/mol)

Lab PM Concentration Reference (μg/mol)

Horiba-1 Horiba-2 Horiba-3 Lab Reference
85

SwRI Project No. 14936.12

FIGURE 51. STEADY STATE SENSORS PEMS PM CONCENTRATION VERSUS THE

LABORATORY REFERENCE

0

50

100

150

200

250

300

350

400

450

500

0 100 200 300 400 500

PEMS PM Concentration (μg/mol)

Lab PM Concentration Reference (μg/mol)

Sensors-1 Sensors-2 Sensors-3 Lab Reference
86

SwRI Project No. 14936.12

FIGURE 52. STEADY STATE AVL PEMS PM CONCENTRATION VERSUS THE 

LABORATORY REFERENCE

0

50

100

150

200

250

300

350

400

450

500

0 100 200 300 400 500

PEMS PM Concentration (μg/mol)

Lab PM Concentration Reference (μg/mol)

AVL-1 AVL-2 AVL-3 Lab Reference
87

SwRI Project No. 14936.12

FIGURE 53. LINEAR REGRESSION CORRELATION BETWEEN PEMS AND LAB

4.9 STEADY STATE PM ERROR SURFACES 

Figure 54, Figure 55, and Figure 56, and show the original concentration deltas that were 

presented to the steering committee on April 2nd, 2009 at SwRI. The Sensors data includes some 

additional points that were added later due to changing the error tolerances of their post 

processor. Each marker on the plot represents one of the j = 1 to M subsets of data divided up by 

concentration value. Each plot shows both the 5th and 95th percentiles which are based on the 

actual data. The error surfaces for the Monte Carlo model were based on the 1st and 99th

percentiles which were extrapolated from the 5th and 95th percentiles assuming a normal 

distribution of the data. 
88

SwRI Project No. 14936.12

FIGURE 54. STEADY STATE CONCENTRATION DELTAS FOR HORIBA

FIGURE 55. STEADY STATE CONCENTRATION DELTAS FOR SENSORS

-200

-150

-100

-50

0

50

100

0 100 200 300 400 500 600

PM Concentration Delta (μg/mol)

Lab Reference Median PM Concentration (μg/mol)

5th % 95th % 50th %

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

0 100 200 300 400 500 600

PM Concentration Delta (μg/mol)

Lab Reference Median PM Concentration (μg/mol)

5th % 95th % 50th %
89

SwRI Project No. 14936.12

FIGURE 56. STEADY STATE CONCENTRATION DELTAS FOR AVL

The steering committee elected to smooth the error surface by not including some of the data 

points that were within the envelope of surrounding points. A similar decision was made on 

some of the error surfaces included in the gaseous measurement allowance program. 

-150

-100

-50

0

50

100

150

0 100 200 300 400 500 600

PM Concentration Delta (μg/mol)

Lab Reference Median PM Concentration (μg/mol)

5th % 95th % 50th %
90

SwRI Project No. 14936.12

Figure 57, Figure 58, and Figure 59 show the final steady state error surfaces for Horiba, 

Sensors, and AVL, respectively. The data presented in these plots has already been processed to 

remove the CVS variability and adjusted to the 1st and 99th percentile from the 5th and 95th

percentile. The lines on the plot represent the final error surfaces as approved by the steering 

committee. The Horiba and AVL error surfaces were accepted by the steering committee during 

the conference call on June 29th, 2009. Some of the Sensors data was reprocessed with different 

tolerances to increase the data yield, so the Sensors error surface was not accepted by the 

steering committee until the July 14th, 2009 meeting in Indianapolis.

FIGURE 57. FINAL STEADY STATE PM ERROR SURFACE – HORIBA

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 100 200 300 400 500

Corrected PM Delta (μg/mol)

CVS Filter PM Concentration (μg/mol)

Delta 1st % Delta 50th % Delta 99th %
91

SwRI Project No. 14936.12

FIGURE 58. FINAL STEADY STATE PM ERROR SURFACE - SENSORS

FIGURE 59. FINAL STEADY STATE PM ERROR SURFACE – AVL

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 100 200 300 400 500

Corrected PM Delta (μg/mol)

CVS Filter PM Concentration (μg/mol)

Delta 1st % Delta 50th % Delta 99th %

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 100 200 300 400 500

Corrected PM Delta (μg/mol)

CVS Filter PM Concentration (μg/mol)

Delta 1st % Delta 50th % Delta 99th %
92

SwRI Project No. 14936.12

With a few exceptions the PEMS generally exhibited a low bias, with the majority of the 50th

percentile deltas falling below zero. The Horiba and Sensors both showed a degree of level 

dependence with the negative bias generally increasing for higher PM concentrations. At a lab 

reference concentration of 413 μg/mol the Horiba 50th percentile was -92 μg/mol and the Sensors 

50th percentile was -177 μg/mol at a lab reference concentration of 403 μg/mol. The AVL 50th

percentile was generally level independent and remained between -8 and -58 μg/mol for the 

entire range of concentrations tested. While the 1st percentile for the AVL also remained 

relatively constant throughout the concentration range, the 99th percentile jumped to a much 

higher value at high concentration. The AVL 99th percentile jumped from 25 μg/mol at 291 

μg/mol to 165 μg/mol at 402 μg/mol. The cluster of positive deltas that caused this large increase 

in the 99th

4.10 TRANSIENT ENGINE RESULTS

percentile data is due solely to mode 2 with PEMS 2. The majority of the positive 

deltas observed for the Horiba were due to mode 4 which was a high speed light load condition. 

Since it is understood that the DCS (EAD) real time particle sensor in the Horiba system is more 

sensitive to smaller particles it was assumed that may have a larger number of sub-50 nm 

particles than the other 5 modes tested, causing more of the filter mass to be attributed to mode 4.

The transient engine testing was designed to characterize the precision error of the PEMS. An 

transient cycle consisting of 30 NTE events was repeated multiple times and any differences in 

the PEMS measurements was attributed to PEMS variability under the assumption that the 

engine operation and PM emissions remained constant. No lab reference value was captured on 

an individual NTE event basis, because it was not possible to measure 30 NTE events with filters 

for each transient cycle. Instead filter measurements were taken for each integrated cycle as a 

general indicator of the PM emissions of the engine. The cycle used in this testing was developed 

using the cycle generator from the gaseous measurement allowance program. In the gaseous 

program the cycle generator was used to develop 20 unique cycles with different orders of NTE 

events and different transitions between events. The steering committee elected to generate one 

single cycle for the current work that would be used for all transient testing. Additional time was 

added in between NTE events to ensure that the PPMD could sample throughout the cycle using 

without missing any NTE events under the assumption of 7 working crystals (6 for sample and 

one reference crystal). In addition several short non-valid NTE events were added to the cycle to 

challenge the PEMS for measurements of non-valid NTE events.

The DPF bypass was adjusted to produce an integrated cycle BSPM of approximately 0.03 g/hphr based on the CVS filter. Instead of readjusting the bypass for each set of PEMS, the exhaust 

valves were set to the same position each time. The NTE events were all approximately 34 

seconds in duration. The original events from the cycle generator were each 32 seconds, however 

the Sensors PEMS would occasionally see these events as shorter than 30 seconds and exclude 

them. Many of the NTE events contained extreme accelerations and decelerations stopping just 

short of the lower boundaries of the NTE window. Considerable time was spent to ensure that 
93

SwRI Project No. 14936.12

the J1939 signal remained in the NTE window during events, although the engine performance 

shifted slightly and occasionally an event was invalidated. If any NTE event did no remain in the 

NTE windows for at least 30 seconds it was not included in the data for the transient error 

surface. Figure 60 shows repeat engine speed traces for the transient cycle. The first and last 

official cycle conducted with each PEMS is shown.

FIGURE 60. REPEAT ENGINE SPEED TRACES FOR NTE TRANSIENT CYCLE

A total of 16 cycles were run for PEMS1, 17 for PEMS2, and 18 for PEMS2. Figure 61 shows 

the repeat torque traces for the transient cycle. The COV on cycle work was 0.7 percent over 51 

cycles.

0

5

10

15

20

25

30

35

600

800

1000

1200

1400

1600

1800

2000

2200

0 500 1000 1500 2000

Engine Speed (rpm)

NTE Event Number

Time (sec)

PEMS1-1 PEMS1-16 PEMS2-1 PEMS2-17 PEMS3-1 PEMS3-18 Event Number
94

SwRI Project No. 14936.12

FIGURE 61. REPEAT ENGINE TORQUE TRACES FOR NTE TRANSIENT CYCLE

The exhaust concentration of soot, as measured by the AVL PEMS is shown in Figure 62. 

0

5

10

15

20

25

30

35

-200

200

600

1000

1400

1800

2200

0 500 1000 1500 2000

Engine Torue (Nm)

NTE Event Number

Time (sec)

PEMS1-1 PEMS1-16 PEMS2-1 PEMS2-17 PEMS3-1 PEMS3-18 Event Number
95

SwRI Project No. 14936.12

FIGURE 62. REPEAT AVL SOOT CONCENTRATION TRACES FOR NTE 

TRANSIENT CYCLE

The measured exhaust concentrations of soot ranged from 5 to 3800 μg/mol (0.2 to 160 mg/m3) 

during NTE operations, with concentrations of 5 to 14 μg/mol (0.2 to 0.6 mg/m3) outside of the 

NTE window. The widest dynamic range observed during an NTE event was for event 9 where 

the initial spike in soot concentration was measured as high as 3800 μg/mol before falling down 

to 55 μg/mol near the end of the event. Figure 63 gives a closer look at events 20 through 23 to 

show the variability of the real-time signal over the three sets of PEMS. 

0

5

10

15

20

25

30

35

0

500

1000

1500

2000

2500

3000

3500

4000

0 500 1000 1500 2000

NTE Event Number

Exhaust Soot Concentration (μg/mol)

Time (sec)

PEMS3-18 PEMS3-1 PEMS2-17 PEMS2-1
96

SwRI Project No. 14936.12

FIGURE 63. AVL SOOT CONCENTRATION DURING NTE TRANSIENT CYCLES, 

EVENTS 20-23

It should be noted that the AVL PEMS is shown here only because it reports the measured soot 

concentration on a second by second basis without additional processing. The Horiba PEMS also 

measures the concentration on a second by second basis, although it is being used in this 

program as a batch sampler rather than a second by second instrument.

18

20

22

24

26

28

30

0

100

200

300

400

500

600

700

800

900

1000

1440 1490 1540 1590 1640

NTE Event Number

Exhaust Soot Concentration (μg/mol)

Time (sec)

PEMS3-18 PEMS3-1 PEMS2-17 PEMS2-1
97

SwRI Project No. 14936.12

FIGURE 64. COMPARISON OF AVL AND HORIBA REAL TIME SIGNALS DURING 

TRANSIENT CYCLE

There were major differences on some of the peak concentrations most notably NTE event 9, 

where the measured AVL concentration was nearly four times higher than the measured Horiba 

concentration. With no reference, the purpose of this figure is only to show that there are 

differences without trying to quantify the accuracy. The Sensors PEMS performs its 

measurements as a batch sampler so that there was no real-time exhaust concentration reported. 

To properly quantify the precision of the PEMS it is important that the reference value remain 

constant otherwise changes in the source PM emissions will be attributed to the PEMS as 

measurement variability. In examining the CVS filter results, it was clear that the engine PM 

emissions varied somewhat during testing. After reviewing the transient PEMS data at the 

January 28th, 2009 meeting at SwRI, the steering committee requested that the idea of a 

correction to the PEMS data be applied to account for changes in the engine performance. A 

correction factor based on the cycle integrated CVS BSPM was presented to the committee 

during the April 2nd

CC = ∑ CC NN

=1

NN∙CC = CCCCVV 

The correction factor is multiplicative and applied to the PEMS data in the following manner:

, 2009 meeting at SwRI and was accepted for use on all of the transient data. 

The correction factor was calculated as follows:

0

20

40

60

80

100

120

140

160

180

0 500 1000 1500 2000

Exhaust Concentration (mg/m3)

Time (sec)

AVL-Soot Horiba-PM
98

SwRI Project No. 14936.12

PP PP , ,jj = CC ∙ PP PP , ,jj

The CVS BSPM along with the correction factor is shown in Figure 65.

FIGURE 65. CVS BSPM AND CORRECTION FACTOR FOR TRANSIENT CYCLE

The correction factor ranged from 0.78 to 1.24. There were two cycles in which the CVS filter 

measurement was void; in these cases the correction factor was set to one. The CVS brake 

specific PM ranged from 34.4 mg/hp-hr to 21.7 mg/hp-hr with an average of 27.2 mg/hp-hr. The 

COV was 11.4 percent when cycles for all three PEMS are included. The COV was 7.2, 6.0, and 

4.8 percent for cycles from PEMS1, 2, and 3, respectively.

While processing some of the AVL data and comparing to the lab, it was discovered that the 

incorrect CVS BSPM value had been used for one of the cycles. The value of 31.9 mg/hp-hr 

from cycle 1956 had also been used for cycle 1965 instead of the correct value of 28.4 mg/hp-hr. 

The entire set of transient CVS data was scrutinized again and no additional errors were found. 

This erroneous correction factor was included in the final transient error surface included in the 

model. The transient error surface was found to shrink for all PEMS by between 1 and 2 percent 

with the correction applied. Given the small change in the outcome no action was taken to 

correct this mistake. The transient error surfaces presented in this report are those used in the 

Monte Carlo model.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

15

20

25

30

35

40

45

50

55

0 10 20 30 40 50 60

Correction Factor

CVS Brake-Specific PM (mg/hp-hr)

Transient Cycle Number

CVS Filter BSPM Correction Factor
99

SwRI Project No. 14936.12

Figure 66, Figure 67, and Figure 68 show the concentration deltas for Horiba, Sensors, and AVL, 

respectively. In the transient testing there was no lab reference to create deltas. Rather the delta 

was calculated from the 50th percentile of the lab data as follows:

Deltai,j = CFi * mPMi,j – 50th

i

Where mPMi,j is the average flow-weighted PM concentration for the jth repeat of the ith NTE 

event. 50th

i is the 50th percentile of mPM for the ith NTE event. CFi is the previously described 

correction factor to adjust for engine variability.

FIGURE 66. HORIBA CONCENTRATION DELTAS FOR TRANSIENT ENGINE 

TESTING

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400

PM Concentration Delta (μg/mol)

Median Flow-Weighted PM Concentration (μg/mol)

5th-50th Percentile 95th-50th Percentile
100

SwRI Project No. 14936.12

FIGURE 67. SENSORS CONCENTRATION DELTAS FOR TRANSIENT ENGINE 

TESTING

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400

PM Concentration Delta (μg/mol)

Median Flow-Weighted PM Concentration (μg/mol)

5th-50th Percentile 95th-50th Percentile
101

SwRI Project No. 14936.12

FIGURE 68. AVL CONCENTRATION DELTAS FOR TRANSIENT ENGINE TESTING

The 5th and 95th percentiles of the Horiba and AVL PEMS were bounded by plus and minus 50 

μg/mol. The Sensors PEMS had slightly larger deltas extending just beyond 100 μg/mol for the 

95th percentile and -150 μg/mol for the 5th percentile. The bias from the data was removed by 

subtracting the 50th from the 5th and 95th percentiles.

The steady state error surface was designed to quantify the accuracy of the PEMS while the 

transient error surface was designed to quantify the precision of the PEMS. However, a portion 

of the PEMS precision error is inherently captured in the steady state error surface. At the 

meeting on July 15th, 2009 in Indianapolis four different approaches were presented to the 

steering committee for removing the steady state contribution to the precision error from the 

transient error surface. The steering committee elected to proceed with approach 3; for 

simplicity the other three approaches will not be described.

In this approach, the data from all three PEMS from the same manufacturer were pooled together 

(ie Sensors1, Sensors2, and Sensors3). This resulted in a total of 18 steady state data points and 

90 transient data points. A median and a MAD value were calculated for each of the 108 data 

points and used to calculate a MAD relative error, transient effect, root mean squared or 

MADre,tr,rms. The MADre,tr,rms 

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400

PM Concentration Delta (μg/mol)

Median Flow-Weighted PM Concentration (μg/mol)

5th-50th Percentile 95th-50th Percentile

is defined as follows:
102

SwRI Project No. 14936.12

MM , , = �1

NN�� MM ,MM ,

�

NN 2

=1

− 1

NN�� MM ,MM ,

�

NN 2

=1

Where tr denotes transient and ss denotes steady state. Using an rms value eliminates the need 

for estimating the steady state variability at each of 90 transient data points. The error surface 

was defined as the 90 percent confidence interval around zero (or no bias) as:

5 h PP = −1.65 ∙ PPPP ∙ MM , ,

95 h PP = 1.65 ∙ PPPP ∙ MM , ,

Effectively the error surface is a straight line with a slope of +/- 1.65·MADre,tr,rms 

Figure 69

and an intercept 

of zero. The transient error surface is shown in for Horiba, Figure 70 for Sensors, and

Figure 71 for AVL.

FIGURE 69. FINAL HORIBA TRANSIENT PM ERROR SURFACE

-120

-80

-40

0

40

80

120

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile
103

SwRI Project No. 14936.12

FIGURE 70. FINAL SENSORS TRANSIENT PM ERROR SURFACE

FIGURE 71. FINAL AVL TRANSIENT PM ERROR SURFACE

-120

-80

-40

0

40

80

120

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile

-120

-80

-40

0

40

80

120

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile
104

SwRI Project No. 14936.12

The median flow weighted PM concentration on the x-axis of each graph is extended to the 

highest measured concentration observed by any of the PEMS during transient testing. The slope 

of the transient PM error surface is 0.083 (1.41) for Horiba, 0.228 (3.27) for Sensors, and 0.099 

(1.66) for AVL. The 1st percentile for the PEMS are -34, -92, and -40 μg/mol at the maximum 

PM concentrations observed by each of the PEMS during steady state testing (around 400 

μg/mol) for the Horiba, Sensors, and AVL systems. The 1st

4.11 CE-CERT MOBILE LAB CORRELATION

percentiles during steady state testing 

at the same PM level were -177, -366, and -110 μg/mol for the Horiba, Sensors, and AVL, 

respectively. Because the steady state error surface was much larger than the transient it tended 

to dominate the model results as discussed later.

The test planned called for the validation of the model to be performed by the mobile emissions 

laboratory (MEL) operated by the University of California at Riverside Bourns College of 

Engineering Center for Environmental Research and Technology (CE-CERT). The mobile 

laboratory consisted of a full-flow CVS system inside the trailer of a Class A truck. The mobile 

lab was capable of measuring gaseous emissions and filter based PM. The mobile lab was arrived 

at SwRI on April 9th

Figure 72

, 2009 to compare brake-specific PM emissions and ensure that the reference 

during in-use validation is similar to the reference used during laboratory testing. The CE-CERT 

MEL was parked behind the SwRI test cell and an exhaust transfer line was fabricated to allow 

the MEL to measure the full engine exhaust in the same way as the SwRI test cell. Because the 

CVS measurement technique requires the full flow of engine exhaust for emissions 

measurement, the exhaust system was designed in such a way that the exhaust pipe could easily 

be switched between the SwRI CVS and the MEL CVS using the same length and geometry of 

exhaust pipe. shows the exhaust system used for the CE-CERT correlation.

FIGURE 72. EXHAUST CONFIGURATION FOR CE-CERT CORRELATION

To CE-CERT CVS Tunnel

To SwRI CVS Tunnel
105

SwRI Project No. 14936.12

The MEL was set to the same CVS flow, secondary dilution ratio, and filter face velocity as the 

SwRI CVS to make the measurements as close as possible. One key factor that could not be 

controlled was the dilution air for the MEL CVS. The SwRI CVS was able to draw flow that is 

conditioned and maintained between 20 and 30oC. The MEL CVS drew its dilution air from the 

ambient which prevented the control of temperature and relative humidity of the dilution air. 

Each sampling system was conditioned by sampling for 10 hours during steady state DPF engine 

operation at a high exhaust temperature. Active DPF regeneration occurred during this 

conditioning period. A total of 16 short NTE transient cycles were conducted using each CVS 

system. The short NTE transient cycle was a modified version of the NTE transient cycle used 

for official transient PEMS testing. The short cycle included 16 of the original 30 NTE events 

and lasted 755 seconds compared to 2130 seconds for the full NTE cycle. Table 18 shows the 

order of testing. 

TABLE 18. TEST PROCEDURE FOR CE-CERT CORRELATION

A manually triggered active DPF regeneration was performed before the start of each set of four 

test cycles to maintain a similar PM loading level on the DPF. Each day the test order was 

switched so that the SwRI tests were first on days 1 and 3 and the CE-CERT tests were first on 

days 2 and 4. Each test was conducted as a hot-start with a 20 minute hot soak in between test 

cycles or in between the DPF regeneration and the first test cycle. Figure 73 shows the brakespecific PM results from the 16 cycles.
106

SwRI Project No. 14936.12

FIGURE 73. BRAKE-SPECIFIC PM RESULTS FROM CE-CERT CORRELATION

The data from test 9 for CE-CERT was removed due to a filter weight that was deemed to be an 

outlier. The average SwRI BSPM was 0.0287 g/hp-hr with a COV of 5.2 percent based on 16 

repeats. The average CE-CERT BSPM was 0.0265 g/hp-hr with a COV of 3.5 percent based on 

15 repeats. The reported CE-CERT emissions were on average 7.7 percent lower than the SwRI 

reported emissions. The average reported BSCO2

A series of tunnel blanks were measured from both systems over sample periods of 15, 30, and 

60 minutes. 

by CE-CERT was 2.6 percent lower than 

average SwRI reported value. One possible source of discrepancy between the two systems was 

the heat loss in the exhaust pipe prior to its entrance into the CVS. The SwRI system was 

completely sheltered within the test cell, while the CE-CERT exhaust pipe was protruding 

outside in such a way that it was expose to wind and ambient temperature effects. This could 

have resulted in a higher thermophoretic deposition of particles inside the exhaust pipe resulting 

in lower emissions for the CE-CERT system. However, the test plan stated that agreement within 

ten percent was considered sufficient, so the correlation was considered complete and the issue 

was not investigated further.

Figure 74 shows the filter weight gains as a function of sample time.

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Brake-Specific PM (g/hp-hr)

SwRI CE-CERT

Day 1 Day 2 Day 3 Day 4
107

SwRI Project No. 14936.12

FIGURE 74. CVS FILTER WEIGHT GAIN DURING TUNNEL BLANKS

The filter weight gains during tunnel blank operation were similar for the SwRI and CE-CERT 

tunnels. The weight gains ranged from 1.9 to 12.6 μg and generally decreased as the sampling 

time increased. The filter weight gains during the correlation testing were greater than 300 μg 

making any differences in the tunnel blanks insignificant.

4.12 INVESTIGATION OF DPF REGENERATION

The steering committee requested that screening tests be performed on measurements of active 

DPF regeneration. Although the majority of the time active DPF regeneration is excluded from 

in-use measurement as the regulations are currently written there are certain situations where 

regeneration could be included in a valid NTE event. In addition, if the Horiba system were to 

trigger filter sampling during DPF regeneration this would still be included in their calibration 

factor. So even if all NTE events containing active DPF regeneration were considered invalid, 

the measurement accuracy of the Horiba system during valid NTE events could still be affected 

through the filter calibration. For this testing the engine was operated at a medium speed medium 

load condition at steady state and the DPF was allowed to accumulate PM until the ECM 

automatically triggered a regeneration. The valves in the bypass were closed forcing a large 

majority of the exhaust through the DPF although the bypass was not completely sealed. At the 

point when the ECM indicated it was preparing for an active regeneration the PEMS and CVS 

filter were triggered to sample. The PEMS sampled 40 seconds on, five seconds off throughout 

0

2

4

6

8

10

12

14

0 15 30 45 60 75

Filter Weight Gain (μg)

Sample Time (min)

SwRI CE-CERT
108

SwRI Project No. 14936.12

the regeneration, while the CVS filter sampled continuously. Table 19 lists the brake-specific 

PM results of the PEMS and the CVS.

TABLE 19. PM EMISSIONS RESULTS FROM ACTIVE DPF REGENERATION

It is likely that only a small portion of the emissions during the regeneration even included 

elemental carbon as shown by the fact that the AVL PEMS measured only three percent of the 

emissions measured by the CVS filter. The Horiba and Sensors PEMS both measured BSPM 

values on the same order as the CVS filter although the event-by-event emissions were much 

different. The EEPS also sampled continuously from the CVS through a long residence time 

secondary dilution tunnel with a nominal secondary dilution ratio of 2. The EEPS number 

concentration was converted to a mass concentration assuming spherical particles with a density 

of 1 g/cm3. A comparison of the event-by-event emissions of the three PEMS and the EEPS are 

shown in Figure 75.

FIGURE 75. BRAKE SPECIFIC PM EMISSIONS DURING ACTIVE REGENERATION

Although the average brake-specific PM value from the Sensors system was very close to the lab 

value it is unlikely that the Sensors instrument was properly capturing the behavior of the 

regeneration. The event-by-event emissions from the Horiba system agreed well with the EEPS 

CVS Filter Horiba Sensors AVL

No. of Samples 1 36 31 35

Avg BSPM, mg/hp-hr 6.8 4.6 7.1 0.2

0

2

4

6

8

10

12

14

16

18

0 20 40 60 80 100

Brake-Specific PM (mg/hp-hr)

Regeneration Time (%)

Horiba Sensors AVL EEPS
109

SwRI Project No. 14936.12

real-time measurement. Comparing the Horiba measurements to the CVS filter and the EEPS 

indicate that it was more accurate than the other PEMS at measuring the regeneration emissions. 

However, one aspect of the Horiba measurement that was not captured in this experiment, and 

that is the change in sensitivity of the DCS particle instrument to different size particles. The 

filter calibration constant for the regeneration event indicated that the DCS was approximately 

15 times more sensitive to the particles emitted during the active regeneration compared to the 

particles emitted during steady state engine testing. This means that if a steady state engine cycle 

was sampled onto the same filter as the DPF regeneration the system would be much less 

accurate. The Horiba system would over predict the emissions during the regeneration and under 

predict the emissions during normal engine operation. These findings were presented to the 

steering committee at the December 11th

4.13 INVESTIGATION OF STORAGE AND RELEASE

, 2008 meeting in San Antonio. Although this could be a 

major issue in the accuracy of the measurement, due to budget limitation, the steering committee 

did not add DPF regeneration testing to the program given that the regulated aspect of 

regeneration during in-use testing was vague. No data from the DPF regeneration investigation 

was included in the model.

At the same time of the DPF regeneration investigation, the steering committee also request that 

screening tests be performed in the area of storage and release of nanoparticles in the 

aftertreatment system. It has commonly been observed that volatile material emitted by the 

engine at low temperature will deposit on the DPF only to be released quickly when the engine 

exhaust temperature climbs. Three different DPF loading conditions were tested: 

• Low idle: 650 rpm, 65 Nm

• Medium idle: 1200 rpm, 135 Nm

• High idle: 1800 rpm, 135 Nm

idling times of 20, 60, and 90 minutes were tested at each condition. Following the period of low 

temperature loading, the engine was immediately brought to a high temperature condition to 

promote the release of the stored particles. Each of the three idling conditions were tested with 

two high temperature conditions:

• Peak torque: 1200 rpm, 2170 Nm

• Near rated power: 1800 rpm, 1425 Nm

These tests were conducted as a screening exercise with measurements by the EEPS to determine 

which combination produced the largest release of nanoparticles to use for the PEMS testing. 

However none of the tested conditions resulted in any significant particle emissions on a number 

or mass basis. To ensure that the EEPS was not missing something that might have been 

captured by one of the PEMS, two tests were run with the PEMS: low idle to peak torque and 

high idle to peak torque. Only the low idle test is shown here because the results are very similar. 
110

SwRI Project No. 14936.12

The engine was allowed to idle for one hour before going to peak torque for ten minutes, this 

process was repeated three times consecutively for a cycle length of 3.5 hours. The PEMS only

sampled during the peak torque portion of the cycle with the same 40 seconds sample, five 

seconds off cycling used in the DPF regeneration study. 

FIGURE 76. TOTAL EXHAUST NUMBER CONCENTRATION DURING STORAGE 

AND RELEASE

A spike of just over 2.0E6 particles/cm3 was observed during the transition from idle to peak 

torque although this appears to be due to acceleration and a possible slight misalignment between 

the dilution ratio and EEPS measurement rather than a release of particles from the 

aftertreatment. The spike was less than 1.0E6 for the second and third transitions to peak torque. 

Figure 77 shows the brake-specific PM emissions measured by the three PEMS during the peak 

torque portion of the storage and release cycle. All samples are taken during the three repeats of 

the ten minutes at peak torque and no emissions from the idle portion of the cycle are 

represented.

0.0E+00

5.0E+05

1.0E+06

1.5E+06

2.0E+06

2.5E+06

0

500

1000

1500

2000

2500

0 2000 4000 6000 8000 10000 12000

Exhaust Number Concentration (part/cm3)

Engine Speed (rpm), Torque (Nm)

Time (sec)

Speed RPM Torque N-m EEPS
111

SwRI Project No. 14936.12

FIGURE 77. BRAKE-SPECIFIC PM EMISSIONS DURING STORAGE AND RELEASE 

INVESTIGATION

It is unclear why the Sensors was reporting emissions much higher than the other two PEMS. A 

single CVS filter was sampled for the entire ten minutes at peak torque for each repeat for a total 

of three filter measurements. The BSPM from the CVS filter was only between 0.3 and 0.4 

mg/hp-hr for each of the three repeats. This was lower than any of the PEMS and an order of 

magnitude lower than the Sensors PEMS. The three negative emissions were attributed to the 

same crystal which was unusually noisy during the test. A comparison of the average BSPM 

values for the cycle are shown in Table 20.

TABLE 20. AVERAGE BRAKE-SPECIFIC EMISSIONS DURING STORAGE AND 

RELEASE CYCLE

The effect of storage and release on the PEMS emissions was not clear because of the inability to 

generate a significant release of nanoparticles, but it was evident that investigating this 

phenomenon using this experimental configuration was not worthwhile. These findings were 

presented at the December 11th

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

10.0

12.0

0 10 20 30 40

Brake-Specific PM (mg/hp-hr)

Sample Number

Horiba Sensors AVL

1st Repeat 2nd Repeat 3rd Repeat

, 2008 meeting in San Antonio and the steering committee 

declined to pursue any further work in this area. No data from the storage and release 

investigation was included in the model.

CVS Filter Horiba Sensors AVL

No. of Samples 3 39 36 39

Avg BSPM, mg/hp-hr 0.38 0.54 2.97 0.41
112

SwRI Project No. 14936.12

4.14 ENGINE MANUFACTURERS TORQUE AND FUEL ERROR SURFACES

The OEM supplied torque error surface was updated from the gaseous PEMS program and the 

OEM supplied BSFC error surface was replaced with a fuel flow error surface. Five different 

engine manufacturers supplied data from 61 different engines. In addition data was used from the 

four engines tested in the ACES program for a total of 2,099 data points from 65 engines. The 

ECM torque deltas were normalized by the maximum ECM torque. The torque deltas are shown 

in Figure 78.

FIGURE 78. OEM SUPPLIED TORQUE ERRORS

The errors as a percentage of the maximum ECM torque are relatively constant throughout the 

entire measured range indicating that the error as a percentage of point would increase as the 

absolute torque decreases. The much smaller data set used for the gaseous measurement 

allowance program showed constant errors as a percentage of point rather than a percentage of 

maximum. The plot of ECM fuel flow errors as a percentage of maximum ECM fuel flow is 

shown in Figure 79.

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

0.0 0.2 0.4 0.6 0.8 1.0 1.2

(ECM Torque - Lab Torque)/Max ECM 

Torque

ECM Torque/Max ECM Torque
113

SwRI Project No. 14936.12

FIGURE 79. OEM SUPPLIED FUEL FLOW ERRORS

The fuel flow errors were also constant in relation to max fuel flow rate above 20 percent fuel 

flow. Since the fuel flow is expected to remain well above 20 percent during NTE event 

operation, the smaller errors at low fuel flows were not considered important. Although the OEM 

supplied torque and fuel flow error surfaces were created as percentages of point for the gaseous 

measurement allowance, these surfaces were generated as percentages of maximum for the 

current program. The 1st, 50th, and 99th

Table 21. 

OEM Error Surface Deltas for Torque and Fuel Flow

percentiles were calculated for torque and fuel flow and 

sampled in the model using a normal distribution. The error surface deltas are shown in 

TABLE 21. OEM ERROR SURFACE DELTAS FOR TORQUE AND FUEL FLOW

-30%

-20%

-10%

0%

10%

20%

30%

0.0 0.2 0.4 0.6 0.8 1.0 1.2

(ECM Fuel - Lab Fuel)/Max ECM Fuel

ECM Fuel/Max ECM Fuel

1st, 

% point

50th, 

% point

99th, 

% of point

Torque -7.6 -1.7 4.1

Fuel Flow -4.5 -0.1 4.9

Percentiles

Parameter
114

SwRI Project No. 14936.12

5 ENVIRONMENTAL TESTING AND RESULTS

A series of tests were conducted to characterize the PEMS response under in-use conditions such 

as changes in pressure, temperature, and humidity, as well as their response to electromagnetic 

and radio frequency interference and shock and vibration.

Initially the test plan called for the EPA PM Generator system to be used as the PM source for all 

environmental testing. The PM generator is capable of producing soot, volatile hydrocarbons, 

and sulfuric acid to simulate the PM emitted from a diesel engine. However, after the system was 

operated at SwRI it was discovered that the PM generator was too tall to fit into the altitude 

chamber. It was likely that it would not be possible to operate the PM generator under 

atmospheric conditions while the output was subject to a constantly varying pressure. The 

diffusion rate of the hydrocarbon vials is pressure dependant so it would not have been possible 

to maintain a constant PM source while varying the pressure inside of the chamber. SwRI 

proposed using a Jing mini-CAST soot generator in place of the PM generator. The soot 

generator is only a fraction of the size and much easier to operate compared to the PM generator. 

The steering committee agreed to allow the soot generator to be used for the altitude testing but 

requested that the PM generator be used for the temperature and humidity testing. 

It was decided to operate the Electromagnetic Interference / Radio Frequency Interference 

(EMI/RFI) and shock and vibration testing as screening. In this case screening testing meant that 

the PEMS were operated while sampling zero air to look for potential problems. The results from 

the screening testing would then be presented to the steering committee to decide whether to 

proceed with official testing to generate an error surface. A good portion of the motivation to 

conduct the EMI/RFI and vibration testing as screening was the result of the finding of the 

gaseous measurement allowance program that in most cases a failure mode of the PEMS was 

observed only as a malfunction of the system in which it could no longer operated. It was not 

commonly observed that the accuracy of the PEMS was affected while it continued to measure 

without detected problems. 

SwRI’s Mechanical and Material Engineering Division (Division 18) performed the 

environmental testing. The altitude, temperature and humidity testing was performed by Rick 

Pitman and Mike Negrete. The EMI/RFI testing was performed by David Smith and Herbert 

Walker. The shock and vibration testing was performed by David Smith, Mike Negrete, and 

Mark Orlowski. 

5.1 REFERENCE MEASUREMENT TESTING

An eight hour baseline measurement was performed for comparison to the eight hour 

environmental tests. Unlike the gaseous measurement allowance program it was not possible to 

compare the accuracy of the PM-PEMS over the measurement period, only the variability. The 
115

SwRI Project No. 14936.12

PM and soot generators provide a particle source, but the correct concentration of the source was 

unknown. 

During the eight hours of testing the PM concentration and dilution ratio was cycled to allow a 

more accurate assessment of the PEMS performance over a range of operating conditions. Three 

PM levels and four dilution ratios were sampled for a total of 12 test conditions. Figure 80 shows 

the schedule of target PM concentration and dilution ratio for one hour of environmental testing.

FIGURE 80. TARGET DILUTION RATIO AND PM LEVEL PROFILE FOR 

ENVIRONMENTAL TESTING

Each level was sampled at dilution ratios of 6, 12, 20, and 30 for the Horiba and Sensors PEMS. 

The AVL PEMS was maintained at its constant dilution ratio of 5. The PEMS were maintained 

at a target dilution ratio for five minutes. With 35 seconds remaining, the sample trigger was 

enabled for 30 seconds. This allowed for four minutes and 25 seconds for stabilization, 30 

seconds for sampling, and five seconds after sampling to ensure sampling on all PEMS had 

stopped before the target dilution ratio was changed. The PEMS were cycled through the four 

dilution ratios at a single PM concentration level before the process was repeated at the next 

concentration level. It took one hour to cycle through each combination of dilution ratio and PM 

concentration. This profile was repeated eight times for a total test time of eight hours. The 

schematic showing the experimental setup can be seen in 

0

1

2

3

4

0

5

10

15

20

25

30

35

0 10 20 30 40 50 60

Target PM Level

Target Dilution Ratio

Time (min)

Target DR Target PM
116

SwRI Project No. 14936.12

FIGURE 81. EXPERIMENTAL SETUP FOR ENVIRONMENTAL TESTING

The particles from the soot generator passed through a Krypton-85 neutralizer to bring the charge 

of the particles to a distribution similar to engine exhaust. A large volume was placed 

downstream of the neutralizer to make minimize the pressure fluctuations observed by the soot 

generator and also to smooth out any changes in concentration from the generator. One of the

AVL PEMS units was placed outside of the environmental chamber upstream of the orifice and 

overflow so that it was isolated from the chamber conditions. This PEMS served as a reference 

to verify that the soot concentration from the generator was stable. The AVL unit was chosen 

because it could provide a real time measurement of the soot concentration. This was not a 

guarantee that the total PM concentration from the generator was steady, but it typically the 

volatile emissions from the generator would not fluctuate significantly without some change in 

the soot concentration.

The mini-CAST soot generator was used as the particle source for the baseline testing. The 

number mean diameter was approximately 70 nm based on measurement with the EEPS. The 

three concentration levels were nominal concentration levels of 1, 3, and 5 mg/m3 with 

approximately 30 percent organic carbon based on the OC/EC measurement.
117

SwRI Project No. 14936.12

FIGURE 82. HORIBA ENVIRONMENTAL BASELINE MEASUREMENTS

-20

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Measured PM Concentration (μg/mol)

Observation Number
118

SwRI Project No. 14936.12

FIGURE 83. SENSORS ENVIRONMENTAL BASELINE MEASUREMENTS

FIGURE 84. AVL ENVIRONMENTAL BASELINE MEASUREMENTS

-20

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Measured PM Concentration (μg/mol)

Observation Number

-20

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Measured PM Concentration (μg/mol)

Observation Number
119

SwRI Project No. 14936.12

FIGURE 85. REFERENCE MSS ENVIRONMENTAL BASELINE MEASUREMENTS

Although the soot concentration from the generator was likely stable based on the measurements 

from both AVL PEMS it is not clear whether the total PM concentration remained constant 

throughout the test. The Horiba PEMS exhibited a clear downwards trend in measured 

concentration as the test progressed. The reported PM concentration from the Sensors PEMS 

appeared to be somewhat independent of the actual PM concentration sampled. The Sensors data 

was too scattered to either confirm or disconfirm the PM trend observed in the Horiba data. In 

addition to the general downwards trend of the concentration, the Horiba data appeared to 

suggest that the accuracy of the dilution ratio was playing a role in the measurement. In 

observations 25 through 96, the reported concentration decreased each time the dilution ratio 

target increased. This suggests that the Horiba PEMS is has either a positive error on lower 

dilution ratios or a negative error on higher dilution ratios. 

The PEMS and reference AVL were both able to resolve the differences between the three PM 

levels clearly even showing similar responses to small changes in concentration. Figure 86

depicts the relationship between the AVL reference measurement and the AVL PEMS 

measurement.

0

20

40

60

80

100

120

140

Measured PM Concentration (μg/mol)

0 20 40 60 80 100

Observation Number

Ref MSS PEMS MSS
120

SwRI Project No. 14936.12

FIGURE 86. REFERENCE AVL VERSUS PEMS AVL FOR ENVIRONMENTAL 

BASELINE

The correlation coefficient between the two measurements is excellent, better than 0.99. The 

slope of 0.78 is likely to be a combination of difference in response of the two instruments and 

line losses between the two points of measurement. 

5.2 PRESSURE CHAMBER TESTING

The SwRI altitude chamber is capable of simulating altitudes up to 19.8 km. The chamber is 1.5 

m in diameter and 2.1 m tall. Typically the simulated altitude chamber at SwRI is operated only 

under vacuum to simulate altitudes greater than that of San Antonio. The gaseous measurement 

allowance test plan called for pressures up to 101.87 kPa or 45 meters below sea level. The 

elevation of San Antonio is approximately 240 meters above sea level with a barometric pressure 

near 99 (98.4 based on altitude) kPa. In the gaseous measurement allowance program, the 

altitude chamber underwent significant alterations to achieve positive pressures and still many 

problems were encountered. The SwRI engineer in charge of the altitude chamber requested that 

only negative pressures be tested to preserve the integrity of their test equipment. Simply 

changing the positive pressures to ambient would have resulted in a large portion of the testing 

conducted at normal atmospheric pressure. Instead a slight negative pressure of 94.3 kPa (610 m, 

2,000 ft) was repeated twice, once at 1.8 hours and once at 7.2 hours. Figure 87 shows the 

y = 0.7762x

R2 = 0.9991

0

20

40

60

80

100

120

0 20 40 60 80 100 120 140

AVL PEMS Concentration (μg/mol)

AVL Reference Concentration (μg/mol)
121

SwRI Project No. 14936.12

original pressure profile from the test plan as well as the modified profile that was used during 

testing.

FIGURE 87. ORIGINAL AND REVISED PROFILE FOR ALTITUDE TESTING

Significant efforts were devoted to ensuring that a stable PM source could be generated that was 

insensitive to pressure. Because the soot generator contains an open flame operating at 

atmospheric pressure, the properties of the flame and hence the particle generation tended to 

change with the pressure of the outlet. By placing an orifice in the transfer line between the soot 

generator and the pressure chamber it was possible to operate the soot generator at a higher 

pressure and maintain a constant pressure through adjustment of an overflow valve upstream of 

the orifice. Adjustments were only necessary during the pressure ramps to 82.7 kPa and 90.0 

kPa. The valve was adjusted to maintain a constant pressure upstream of the orifice. Figure 88

shows the setup outside of the altitude chamber.

-800

-300

200

700

1200

1700

80

85

90

95

100

105

110

0 1 2 3 4 5 6 7 8

Altitude (m)

Pressure (kPa)

Time (hr)

Original Revised
122

SwRI Project No. 14936.12

FIGURE 88. ALTITUDE TESTING CHAMBER

During the first practice run the AVL PEMS 3 blew a fuse in its measurement unit. The test was 

stopped and the fuse was replaced, but immediately blew again indicating an electrical failure 

within the unit. The unit was replaced with AVL 3 and testing continued with the problem not 

observed again. Due to space constraints, the Semtech DS and Horiba OBS-2200 gaseous PEMS 

were located outside of the environmental chamber. Since the purpose of these devices was only 

communications, it was not considered necessary to test them inside the chamber. The external 

compressor used for the Horiba dilution air was also installed outside the chamber. A large 

compressor was supplied by SwRI to provide oil-less dilution air. The steering committee had 

requested that the Horiba supplied compressor should be used but then agreed to allow the 

replacement compressor after it was determined that the original compressor could not operate 

for the entire eight hours without shutting off. A picture of the PEMS installed in the altitude 

chamber is shown in Figure 89. 
123

SwRI Project No. 14936.12

FIGURE 89. PEMS INSTALLED IN THE ALTITUDE CHAMBER

As mentioned previously, analysis of the environmental data was more difficult for the PMPEMS program compared to the gaseous PEMS program due to the lack of a known reference 

concentration. For this reason, only the variability of the PEMS measurement in comparison to 

its average was compared. The individual data points were plotted as well as the average levels 

to show how far each measurement deviated from the average. Figure 90, Figure 91, and Figure 

92 show the concentration measurements by the Horiba, Sensors, and AVL respectively.
124

SwRI Project No. 14936.12

FIGURE 90. HORIBA ENVIRONMENTAL PRESSURE MEASUREMENTS

The Horiba concentration measurements were not consistent for different dilution ratios. In the 

baseline test it was observed that increasing the dilution ratio in the range of 6, 12, 20, and 30 

tended to decrease the reported concentration. In the pressure test, it appears that the measured 

concentration increases as the dilution ratio increases. The largest deviations from the average 

occurred during observations 53-60 while the pressure was at its lowest indicating that the 

ambient pressure does have some effect on the accuracy of the Horiba reported concentration.

80

82

84

86

88

90

92

94

96

98

100

0

20

40

60

80

100

120

140

0 20 40 60 80 100

Chamber Pressure (kPa)

PM Concentration (μg/mol)

Observation Number

Avg Horiba Horiba Pressure
125

SwRI Project No. 14936.12

FIGURE 91. SENSORS ENVIRONMENTAL PRESSURE MEASUREMENTS

The Sensors data exhibited a large amount of variability with some data more than a factor of 

two higher and lower than the average. With that amount of scatter it was difficult to visually 

discern an effect of pressure on the measurement.

80

82

84

86

88

90

92

94

96

98

100

0

50

100

150

200

250

0 20 40 60 80 100

Chamber Pressure (kPa)

PM Concentration (μg/mol)

Observation Number

Avg Sensors Sensors Pressure
126

SwRI Project No. 14936.12

FIGURE 92. AVL ENVIRONMENTAL PRESSURE MEASUREMENTS

The AVL data was grouped tightly around the average except during the low pressure excursion 

around observances 53-60. Because of the excellent repeatability of the measurement, the effect 

of pressure on the measurement was readily apparent. As the ambient pressure decreased, the 

measurement decreased as well.

To determine whether an error surface would be generated for each PEMS, the MAD from the 

baseline was compared to the MAD of the pressure test. If the MAD of the pressure test was 

greater, an error surface was generated to present to the steering committee. The plots of median 

versus MAD shown in Figure 93, Figure 94, and Figure 95 were presented to the steering 

committee at the meeting in Indianapolis on July 15th

80

82

84

86

88

90

92

94

96

98

100

0

10

20

30

40

50

60

70

80

0 20 40 60 80 100

Chamber Pressure (kPa)

PM Concentration (μg/mol)

Observation Number

Avg AVL AVL Pressure

, 2009.
127

SwRI Project No. 14936.12

FIGURE 93. AVL PRESSURE MEDIAN VERSUS MAD

FIGURE 94. HORIBA PRESSURE MEDIAN VERSUS MAD

y = 0.022x - 0.124

R2 = 0.999

y = 0.047x + 0.539

R2 = 0.413

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0 20 40 60 80 100 120

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

AVL-Baseline AVL-Pressure

y = 0.073x + 5.936

R2 = 0.427

y = 0.121x - 4.006

R2 = 0.991

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

0 20 40 60 80 100 120

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Horiba-Baseline Horiba-Pressure
128

SwRI Project No. 14936.12

FIGURE 95. SENSORS PRESSURE MEDIAN VERSUS MAD

The MAD values were higher for the baseline compared to the pressure test for both Horiba and 

Sensors indicating that no additional variability in the measurement could be attributed to 

changes in pressure using this technique. No environmental error surface was calculated for 

either Horiba or Sensors. The AVL MAD values were higher for the pressure test compared to 

the baseline, so an error surface was developed. It should be noted that the variability of the AVL 

is much lower for the AVL compared to the Horiba and Sensors, however because of the high 

precision of the AVL measurements during the baseline it was still possible to discern the added 

variability due to changes in ambient pressure. The environmental pressure error surface was 

calculated using the same pooled rms technique that was used for the transient error surface. 

Figure 96 shows the error surface that was generated for environmental pressure on the AVL 

PM. The error surface was accepted for use by the steering committee during the July 15th

y = 0.155x + 15.834

R2 = 0.756

y = 0.111x + 15.755

R2 = 0.609

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

0 20 40 60 80 100 120 140

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Sensors-Baseline Sensors-Pressure

, 2009 

meeting in Indianapolis.
129

SwRI Project No. 14936.12

FIGURE 96. FINAL ERROR SURFACE FOR ENVIRONMENTAL PRESSURE AVL PM 

CONCENTRATION

This error surface was similar in magnitude to the AVL transient PM error surface, however it 

was much smaller than the AVL steady state PM error surface.

5.3 TEMPERATURE & HUMIDITY CHAMBER TESTING

The temperature and humidity chamber testing was designed to characterize the effects of 

ambient temperature and humidity on the accuracy of PM measurement. The testing chamber has 

the capability of independently controlling the temperature and moisture content of the air in the 

range of typically observed levels from ambient conditions. The temperature and humidity 

profile from the gaseous program was modified based on data acquired during CE-CERT testing. 

At the June 12th

Figure 97

, 2008 meeting in Madison data was presented by Kent Johnson indicating that 

temperatures near the PEMS instruments could be above 60°C during in-use testing. With this in 

mind the steering committee elected to add temperatures as high as 50°C to the profile, where the 

original profile had a maximum temperature of just over 30°C. The original and modified 

temperature and humidity profiles are shown in .

-12

-8

-4

0

4

8

12

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile
130

SwRI Project No. 14936.12

FIGURE 97. TEMPERATURE AND HUMIDITY PROFILE FOR ENVIRONMENTAL 

TESTING

The temperature chamber was unable to control the humidity under a temperature of 5°C so the 

humidity was uncontrolled during the portion of the cycle where the temperature was between 

2°C and 5°C. The moisture content was quite small during this portion of the cycle, so this was 

not considered to be a significant issue. Figure 98 shows the PEMS in the temperature and 

humidity chamber.

0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

Relative Humidity (%)

Temperature (°C)

Time (hr)

Original Temp Modified Temp Original RH Modified RH
131

SwRI Project No. 14936.12

FIGURE 98. PEMS INSTALLED IN THE TEMPERATURE AND HUMIDITY 

CHAMBER

The compressor pictured was provided by SwRI for use during environmental testing. The 

original compressor supplied by Horiba was much smaller, but also had difficulty completing 

eight hours of operation without shutting off. The compressor was included in the temperature 

chamber so that the dilution air for the Horiba system would be affected by the same changes in 

temperature and humidity that the dilution air of the Sensors and AVL systems would 

experience. The PM sample was transported into the chamber using a heated sample line 

maintained at 60°C. The temperature was set slightly above the maximum temperature of the 

chamber so that a constant temperature in the sample could be maintained throughout the test. 

There was a small portion of the end of the transfer line that was not heated, but it was 

extensively insulated to minimize temperature effects. The EPA PM generator was used in

conjunction with the soot generator in this work to provide the particle source. The experimental 

setup was shown previously in Figure 81. The PM generator is shown in Figure 99.
132

SwRI Project No. 14936.12

FIGURE 99. THE PM GENERATOR INSTALLED OUTSIDE THE TEMPERATURE 

AND HUMIDITY CHAMBER

The Horiba and Sensors gaseous PEMS were installed outside of the chamber along with the PM 

generator. The PM generator is designed to add volatile hydrocarbons, sulfuric acid, and water 

vapor to an elemental carbon source. An oxidation catalyst removed any volatile from the soot 

generator before volatile was added from the diffusion vial ovens. Unfortunately both the sulfur 

oven and the syringes injecting water malfunctioned during testing and neither were able to be 

quickly repaired. For the official temperature testing the particle source consisted of elemental 

carbon from the mini-CAST and volatile hydrocarbon from the PM generator. 
133

SwRI Project No. 14936.12

FIGURE 100. HORIBA ENVIRONMENTAL TEMPERATURE MEASUREMENTS

The Horiba data showed a significant amount of variability although it is unclear from this graph 

whether the variability is directly related to the changing temperature. For example, the 

measurements between observation 25 and 35 when the temperature is below 10°C tend to be 

below the cycle average. However, the second time the temperature drops below 10°C around 

observation 70, the measurements are above the cycle average. The temperature measurements 

for the Sensors PEMS are shown in Figure 101. 

0

10

20

30

40

50

60

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Chamber Temperature (°C)

PM Concentration (μg/mol)

Observation Number

Avg Horiba Horiba Temperature
134

SwRI Project No. 14936.12

FIGURE 101. SENSORS ENVIRONMENTAL TEMPERATURE MEASUREMENTS 

The Sensors data displayed a high degree of variation from the mean with no clear trend related 

to temperature. No valid data was captured for the last hour of operation because the Sensors 

PES was unable to maintain the crystals at a temperature of 50°C when the chamber temperature 

was above 47°C. 

0

10

20

30

40

50

60

0

50

100

150

200

250

0 20 40 60 80 100

Chamber Temperature (°C)

PM Concentration (μg/mol)

Observation Number

Avg Sensors Sensors Temperature
135

SwRI Project No. 14936.12

FIGURE 102. AVL ENVIRONMENTAL TEMPERATURE MEASUREMENTS

The AVL data exhibited excellent repeatability in comparison to the other two PEMS. Any 

temperature dependence by the AVL PEMS was extremely small. The temperature data was 

compared in the same manner as the pressure data with the MAD values plotted in relation to the 

median. 

0

10

20

30

40

50

60

0

20

40

60

80

100

120

140

160

0 20 40 60 80 100

Chamber Temperature (°C)

PM Concentration (μg/mol)

Observation Number

Avg AVL AVL Temperature
136

SwRI Project No. 14936.12

FIGURE 103. HORIBA TEMPERATURE MEDIAN VERSUS MAD

FIGURE 104. SENSORS TEMPERATURE MEDIAN VERSUS MAD

y = 0.073x + 5.936

R2 = 0.427

y = 0.084x + 3.106

R2 = 0.767

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

20.0

0 20 40 60 80 100 120 140 160

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Horiba-Baseline Horiba-Temperature

y = 0.155x + 15.834

R2 = 0.756

y = 0.040x + 22.144

R2 = 0.446

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

0 20 40 60 80 100 120 140 160

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Sensors-Baseline Sensors-Temperature
137

SwRI Project No. 14936.12

FIGURE 105. HORIBA TEMPERATURE AND HUMIDITY MEDIAN VERSUS MAD

The MAD of the Horiba baseline is slightly higher than during the temperature test meaning that 

an error surface was not generated in this case. The Sensors and AVL MAD was higher for the 

baseline at lower concentrations, but higher for the temperature data for higher concentrations. 

An attempt was made to calculate an error surface for Sensors, but the pooled rms technique 

resulted in a slightly higher value for the baseline indicating that an error surface was not 

necessary. The error surface calculated for AVL using the same technique is shown in 

y = 0.022x - 0.124

R2 = 0.999

y = 0.006x + 1.072

R2 = 0.962

0.0

0.5

1.0

1.5

2.0

2.5

0 20 40 60 80 100 120 140 160

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

AVL-Baseline AVL-Temperature
138

SwRI Project No. 14936.12

FIGURE 106. FINAL ERROR SURFACE ENVIRONMENTAL TEMPERATURE AND 

HUMIDITY AVL PM

The PM median concentrations tested were between 25 and 135 μg/mol. It was necessary to 

extend the error surface out to nearly 500 μg/mol to encompass the range of concentrations 

encountered during engine testing. The steering committee elected to cap the error surface at plus 

and minus 5.3 μg/mol because it was unclear whether the errors would continue to increase 

outside of the concentrations observed in the temperature and humidity testing. Extending a 

straight line out to the median concentration of 481 μg/mol would have resulted in a 5th

5.4 ELECTROMAGNETIC AND RADIO FREQUENCY INTERFERENCE 

SCREENING

percentile of 18.9 μg/mol.

The electromagnetic inference (EMI) and radio frequency interference (RFI) testing was 

conducted as a screening exercise without the use of PM source. During the gaseous

measurement allowance program the majority of the problems encountered caused a malfunction 

of the PEMS to the point where it would no longer operate. The main purpose of the 

environmental testing was not to test the durability of the PEMS, but to quantify any errors that 

might occur during the operation of a PEMS that would cause a measurement error. The same 

series of tests from the gaseous EMI/RFI testing were conducted as a series of individual 

screening tests using only HEPA filtered air. Based on the screening results it remained a 

possibility to conduct a full test cycle using a particle source to generate an error surface.

-12

-8

-4

0

4

8

12

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile
139

SwRI Project No. 14936.12

By providing filtered room air to the PEMS it would still have been possible to detect a wide 

range of possible measurement accuracy issues, although some problems may only present 

themselves when a particle sample is present. To shorten the test time, the PEMS were triggered 

continuously so that time in between tests could be minimized. This meant that the Horiba 

system was continuously sampling on a filter, and the Sensors system was continuously sampling 

on a crystal. The Horiba system was able to sample continuously for eight hours or more on the 

filter since no particle source was loading on the filter. However, the Sensors system had a 

maximum sample time that it was adjusted so that it would cycle through the crystals every 120 

seconds. This was desirable since it exercised the operation of all eight crystals however some

issues surfaced due to this testing technique. The problems included: both high voltage power 

supplies turning on at the same time, two crystals sampling at the same time, no crystal sampling 

even though one was available. Each of these problems occurred infrequently, and it was 

believed that the problems were a result of leaving the trigger on continuously for long periods of 

time since none of these issues were observed during engine testing. These were functionality 

issues that did not appear to have any impact on the accuracy of the measurement.

The EMI and RFI testing was conducted in a radiation chamber with walls covered with large 

cones of carbon impregnated foam designed to absorb radiation and minimize reflections. Four 

standard Society of Automotive Engineers (SAE) tests were conducted: Bulk Current Injection, 

Radiated Immunity, Electrostatic Discharge, and Conducted Transients.

The Sensors and AVL PEMS were both designed to run off the vehicle’s 24 Volt electrical 

system so both systems were powered using a 12 V automotive battery that was continually 

charged using a 120 VAC charger supplied by Sensors. The Sensors PEMS was a 12V system so 

it connected directly to the battery. The AVL system is powered by 120 Volts 60 Hz AC power 

so a commercially available inverter was provided by AVL to convert the 12 volts DC into 120 

volts AC. The Horiba system was designed to operate using a generator, therefore it was still 

powered using the 120 VAC wall outlets. 
140

SwRI Project No. 14936.12

FIGURE 107. AVL PEMS IN THE RADIATION CHAMBER FOR EMI AND RFI 

TESTING

5.4.1 Bulk Current Injection

SAE test J1113/4 Immunity to Radiated Electromagnetic Fields – Bulk Current Injection was 

performed to determine the effect of electromagnetic radiation on the electrical cables of the 

PEMS. The specifications used are detailed in Region 2, Class B of the J1113/4 test protocol. A 

calibrated current probe was place around the electrical cable and used to inject RF current into 

the cable. Figure 108 shows a cable running through the bulk current injection probe.
141

SwRI Project No. 14936.12

FIGURE 108. BULK CURRENT INJECTION PROBE

The probe was positioned at 120 mm, 450 mm, and 750 mm from the cable connector to test the 

cable three times. For each test, the frequency of the current was stepped from 1 MHz to 400 

MHz using the following step sizes.

1 MHz to 10 MHz – 1 MHz step size

10 MHz to 200 MHz – 10 MHz step size

200 MHz to 400 MHz – 20 MHz step size

These were the maximum allowed step sizes according to the SAE protocol. As with the gaseous 

program a 5 second dwell time was used to ensure the electromagnetic field had stabilized before 

switching to the next frequency. The probe was calibrated to deliver 60 milliamps of current as 

specified in the test procedure. Figure 109 shows the bulk current injection probe being used to 

test a cable on the Horiba PEMS.
142

SwRI Project No. 14936.12

FIGURE 109. HORIBA PEMS SETUP FOR BULK CURRENT INJECTION

Test Cable BCI Probe
143

SwRI Project No. 14936.12

FIGURE 110. SENSORS SETUP DURING BULK CURRENT INJECTION TESTING

FIGURE 111. AVL SETUP DURING BULK CURRENT INJECTION TESTING
144

SwRI Project No. 14936.12

The cables tested on the Horiba system were the sample line temperature, sample line heater 

power, heated enclosure heater power, sample trigger and exhaust flow signal, ethernet 

connection from the DCS to the electrical enclosure, ethernet connection from the TRPM to 

2200, DCS 12 volt power, and two AC power cords. There were two main problems discovered 

when the bulk current injection was applied to the cables of the Horiba PEMS. When the line 

supplying power to the heater in the heated enclosure was probed, the reported total flow of the 

system dropped started to rapidly fluctuate around the frequency of 10 MHz. This frequency was 

manually repeated after the sweep and the same behavior occurred. Figure 112 shows the 

response of several of the Horiba signals to the current injection.

FIGURE 112. HORIBA FLOW DISTURBANCE FROM BULK CURRENT INJECTION

The noise in the system appeared to have originated with several of the temperature and pressure 

measurements including the filter face temperature and total flow temperature shown here. The 

total flow temperature and pressures began to fluctuate causing fluctuations in the total flow 

measurement, which in turn caused the dilution flow to fluctuate in an effort to maintain 

proportionality. This problem was only observed on the cable supplying heater power to the 

heated enclosure box.

0

10

20

30

40

50

60

70

80

0

5

10

15

20

25

30

35

0 500 1000 1500 2000

Temperature (°C)

Flow (slpm)

Time (sec)

OB-PFSS_Total_1 [L/min] OB-PFSS_Dil_1 [L/min]

OB-PFSS_Filter_T [degC] OB-PFSS_Tot_T [degC]

10 MHz During Sweep 10 MHz Repeated
145

SwRI Project No. 14936.12

The communications between the laptop and the PEMS was disrupted when the Ethernet cables 

were probed. Several times the communications dropped out completely. The problem was 

intermittent around 40 to 50 MHz and 140 to 160 MHz. 

The final major problem experienced by the Horiba system during BCI was a large amount of 

noise in the exhaust flow signal when the analog signal cable between the OBS-2200 and OBSTRPM was probed. The exhaust flow is measured by the Horiba gaseous PEMS OBS-2200. The 

OBS-2200 outputs the exhaust flow measurement as an analog voltage which is then read by the 

OBS-TRPM Over frequencies between 1 and 40 MHz the exhaust flow signal read as high as 

3,000 kg/hr on the OBS-TRPM while the measured value on the OBS-2200 was less than 10 

kg/hr. The exhaust flow signal during the BCI sweep is shown in Figure 113.

FIGURE 113. HORIBA EXHAUST FLOW NOISE ON ANALOG CABLE DURING 

BULK CURRENT INJECTION

It was clear that the problem involved only the transmission of the exhaust flow signal as an 

analog voltage between the two systems, not the actual measurement of the exhaust flow. For in 

use testing this problem would be easily detectable when the two exhaust flow data from the two 

different files are compared; however, the TRPM relies on the analog exhaust flow signal for 

adjusting its dilution ratio for proportional sampling. Errors in the exhaust flow signal would 

cause the TRPM to lose its proportionality, and therefore any data collected with the same filter 

sampling would be voided.

-500

0

500

1000

1500

2000

2500

3000

3500

0 200 400 600 800 1000

Exhaust Flow (kg/hr)

Time (sec)

OBS-TRPM OBS-2200
146

SwRI Project No. 14936.12

The cables tested on the Sensors system were the communication cable from the PPMD to DS, 

and the DC power. As mentioned previously, many of the problems that were encountered with 

the Sensors PEMS happened intermittently and could not be reproduced at any specific 

frequencies. It was assumed that problems involving crystal sampling switching, high voltage 

power supplies, and fluctuations in the corona current were not induced by the bulk current 

injection. However, the Sensors system did experience significant communication problems 

when the serial cable between the PPMD and DS units was probed. The communication was 

repeatedly disrupted over the entire frequency range causing a loss of data. In nearly every case 

the communication was recovered once the radiation subsided allowing normal operation to 

continue.

The cables tested on the AVL system were the communications cable between the measuring and 

conditioning unit, the analog output to the Semtech DS, and two AC power cords. The BCI 

induced a significant amount of noise on the analog signal cable between the AVL MSS and the 

Sensors Semtech DS. The AVL system can log its own data, but the data used officially in this 

program and during in use testing is the data logged by the Semtech DS. Two signals are carried

on the analog output wire: the measured concentration and the dilution ratio. These two numbers 

are multiplied together for the final reported concentration. The measured concentration did not 

see significant noise, but the dilution ratio signal was strong influenced by radiation in the range 

of 1-3 MHz and 40 MHz. A comparison of the MSS measured dilution ratio and the dilution 

ratio recorded by the Semtech DS is shown in Figure 114.
147

SwRI Project No. 14936.12

FIGURE 114. BCI NOISE ON AVL ANALOG OUTPUT CABLE

The dilution ratio doubled from 5 to 10 during current injection at 1-3 MHz. The data in the plot 

was recorded during the frequency sweep, and the problem was replicated manually afterwards. 

Since the true dilution ratio was still recorded in the MSS log file, the correct reported 

concentration could be recovered after testing.

5.4.2 Radiated Immunity

The response of the PEMS to continuous narrowband electromagnetic fields was measured using 

SAE test J1113/21 Electromagnetic Compatibility Measurement Procedure for Vehicle 

Components – Part 21: Immunity to Electromagnetic Fields, 10 kHz to 18 GHz, Absorber-Lined 

Chamber. The exact specifications of the tests performed were drawn from Region 2, Class B of 

the J1113/21 protocol. Several different antennae were used to generated electromagnetic 

radiation over the frequency range of 10 kHz to 1 GHz. The electromagnetic susceptibility 

experts at SwRI recommended ending the test at a frequency of 1 GHz rather than 18 GHz due to 

the very small probability of detecting any susceptibility above 1 GHz. This same approach was 

used in the gaseous PEMS program. The carbon impregnated foam walls of the radiation 

chamber was designed to absorb any radiation so that the PEMS would only see the direct 

radiation generated by the antenna. The following step sizes were used during the frequency 

sweeps:

• 10 kHz to 100 kHz – 10 kHz step size

0

2

4

6

8

10

12

0 100 200 300 400 500

Dilution Ratio

Time (sec)

Semtech DS Analog DR AVL MSS DR

1-3 MHz

40 MHz
148

SwRI Project No. 14936.12

• 100 kHz to 1 MHz – 100 kHz step size

• 1 MHz to 10 MHz – 1 MHz step size

• 10 MHz to 200 MHz – 2 MHz step size

• 200 MHz to 1 GHz – 20 MHz step size

The SAE standard field intensity of 50 volts/meter was used with both vertical and horizontal 

electromagnetic radiation orientations. 

FIGURE 115. HORIBA PEMS SETUP DURING RADIATED IMMUNITY TESTING
149

SwRI Project No. 14936.12

FIGURE 116. SENSORS PEMS SETUP DURING RADIATED IMMUNITY TESTING

FIGURE 117. AVL PEMS SETUP DURING RADIATED IMMUNITY TESTING

The Horiba PEMS experienced several problems related to the radiated immunity test including 

a loss of communications between the laptop and the instrument. This problem occurred at 

several different frequencies for both horizontal and vertical polarizations. Because the data is 
150

SwRI Project No. 14936.12

logged on the laptop any loss of communication is also a loss of data. This would result in a 

voided test since the filter calibration relies on collecting data the entire time the filter is 

sampling. A problem was also observed with the control of the dilution ratio. During EMI and 

RFI testing the Horiba dilution ratio was maintained constant at 6.

FIGURE 118. HORIBA DILUTION RATIO FLUCTUATIONS DURING RADIATED 

IMMUNITY

The dilution ratio varied between 338 and -938 but only a smaller portion of the graph is shown 

for more detail. The external flowmeter on the end of the sample flow indicated that these rapid 

changes in flow were real and not just reported. The problem with the exhaust flow analog 

output signal first encountered during the bulk current injection testing was also observed during 

radiated immunity. 

The AVL system reported a supply voltage error in the range of 200 MHz to 1 GHz of horizontal 

radiation. The inverter powered down at 300 MHz shutting down the system completely.

5.4.3 Electrostatic Discharge

SAE test J1113/13 Electromagnetic Compatibility Measurement Procedure for Vehicle 

Components—Part 13: Immunity to Electrostatic Discharge was performed to test the PEMS 

response to Electrostatic Discharges (ESDs) on surfaces and connectors. The exact procedure 

used was taken from Region 2, Class B of the J1113/13 standard. Approximately 40 ESDs were 

-100

-80

-60

-40

-20

0

20

40

60

80

100

0 500 1000 1500 2000 2500 3000 3500

Dilution Ratio

Time (sec)

Measured Dilution Ratio Setpoint Dilution Ratio

50-70 MHz 

Vertical

120-150 MHz 

Vertical
151

SwRI Project No. 14936.12

supplied to each PEMS at various connector ports, exposed screws, and general surfaces. The 

test included both direct contact discharges as well as indirect discharges with the electrostatic 

discharge gun placed near the surface of interest. The ESD gun is shown in Figure 119.

FIGURE 119. ELECTROSTATIC DISCHARGE SIMULATOR

The ESD gun was calibrated using an electrostatic voltmeter to deliver 4000 volts per discharge. 

Neither the Horiba nor the AVL PEMS exhibited any susceptibility to the ESD tests. The 

Sensors unit shut off completely with an ESD near the auxiliary connector port to which an 

external flow meter connects for audit purposes. This problem was confirmed with a second 

ESD. No issues possibly related to measurement accuracy were found with any of the PEMS.

5.4.4 Conducted Transients

The response of the PEMS to voltage disturbances in the 12 volt power supply cable was 

checked using SAE test J1113/11 Immunity to Conducted Transients on Power Leads. The tests 

were conducted using specifications found in Region 2, Class B of the J1113/11 protocol. A 

Schaffner NSG 5000 Automotive Electronics Test System was installed in between the 12 volt 

power supply and the PEMS. The Schaffner Electronics Test System delivered voltage 

perturbations to the PEMS of varying magnitudes and durations. The voltage spikes ranges from 

-200 to 100 volts and lasted anywhere between 250 ns and 200 ms. The tests included quick 

voltage recovery, slow voltage recovery, repeated voltage bursts, and load dump. The conducted 

transients tests were not performed on the Horiba PEMS, because the Horiba system was 
152

SwRI Project No. 14936.12

intended for use with an external generator and does not use 12 volt power. The Schaffner Test 

System can be seen in Figure 120 with the Sensors PEMS and Figure 121 for the AVL PEMS.

FIGURE 120. SENSORS SETUP DURING CONDUCTED TRANSIENTS TESTING

FIGURE 121. AVL SETUP DURING CONDUCTED TRANSIENTS TESTING

The Sensors PEMS powered down during a -100 volt spike with quick recovery and again 

powered down when the magnitude of the spike was reduced to -50 volts and then -25 volts. The 

slow recovery voltage spikes also caused the PEMS to power down with the shortest dwell time 

of 40 ms causing the unit to shut down. Longer dwell times were not tested.
153

SwRI Project No. 14936.12

The quick recovery voltage spike caused the AVL inverter to power down, although it appeared 

to be working properly when it was restarted. The inverter shut down and started smoking during 

the slow recovery voltage spikes. The inverter would no longer work, and the test was not 

repeated with another inverter to prevent further damage.

The findings from the EMI and RFI testing were presented to the steering committee during the 

May 20th

5.5 VIBRATION TESTING

, 2009 in San Antonio. The steering committee declined to pursue further testing to 

develop any EMI and RFI error surfaces. The steering committee requested that Horiba 

investigate the issue of bulk current injection noise in the analog exhaust flow cable and that 

Sensors investigate the issue of exhaust flow errors during radiated immunity testing.

The vibration testing was conducted as a screening exercise similar the method used for the EMI 

and RFI testing. The PEMS were operated on HEPA filtered room air while being subjected to 

vibration. The response of the PEMS to the screening exercise was used to determine whether 

further tested was needed to generate an error surface for vibration. Each PEMS was mounted on 

an Unholtze-Dickie Shaker Table which was capable of vibrating the PEMS on all three axes 

separately by adjusting the orientation of the PEMS. The system used a large table to vibrate 

horizontally. By rotating the orientation of the PEMS, this table was able to simulate longitudinal 

and transverse horizontal vibration. The shaker was rotated into a vertical position and a smaller 

platform was attached to provide vertical vibration. Due to considerations for non-road in-use 

testing, the steering committee requested that the PEMS also be rotated at a 45 degree incline on 

each axis as well. The horizontal vibration platform was large enough to include all of the pieces 

any one type of PEMS at a time. The vertical platform and 45 degree stand were each capable of 

installing only a single piece of the instrumentation at a time.
154

SwRI Project No. 14936.12

FIGURE 122. SENSORS PEMS MOUNTED FOR TRANSVERSE HORIZONTAL

VIBRATION

FIGURE 123. SENSORS PEMS MOUNTED FOR LONGITUDINAL HORIZONTAL 

VIBRATION

Direction of vibration
155

SwRI Project No. 14936.12

FIGURE 124. SENSORS PEMS MOUNTED FOR TRANSVERSE 45° VIBRATION

FIGURE 125. SENSORS PEMS MOUNTED FOR LONGITUDINAL 45° VIBRATION

Direction of vibration

Direction of vibration
156

SwRI Project No. 14936.12

FIGURE 126. SENSORS PEMS MOUNTED FOR VERTICAL VIBRATION

FIGURE 127. SENSORS PEMS MOUNTED FOR 45° VERTICAL VIBRATION

Direction of vibration

Direction of vibration
157

SwRI Project No. 14936.12

The power spectral density (PSD) from the Mil Standard 810, US Highway Truck Vibration 

Exposure was used in this testing. To prevent damage to the PEMS, the PSD was only operated 

at 25%, 50%, and 75% of the energy specified in the Mil Standard 810. After a few tests at the 

75% level, the energy levels were reduced to 10%, 25%, and 50% to maintain the integrity of the 

instruments. In addition each energy level was only tested for 5 minutes to minimize the chances 

of damaging the instruments. Figure 128 shows the PSD used for vertical vibration testing and 

Figure 129 shows the PSD used for horizontal testing.

FIGURE 128. POWER SPECTRAL DENSITY FOR VERTICAL VIBRATION TESTING
158

SwRI Project No. 14936.12

5.5.1.1

FIGURE 129. POWER SPECTRAL DENSITY FOR HORIZONTAL VIBRATION 

TESTING

Because the OBS-TRPM included three separate boxes, some of the vibration testing had to 

performed on individual boxes. The DCS, HE, and MEEE were all able to fit on the horizontal 

vibration table however only a single box could fit at a time on the 45 degree angle stand. The 

DCS and MEEE were tested on this stand; the HE box was not tested at 45 degrees, because it 

was felt that no errors would be detected in a box housing only a filter holder and a cyclone. A 

particle source would need to be present to detect problems in this portion of the system. The 

vertical vibration platform was able to hold both the DCS and the MEEE, so these two boxes 

were tested simultaneously while the HE was again not tested. Any piece of the system that was 

not being tested on the vibration stand was positioned directly next to the stand so that all flow 

and electrical cables could still connect to the system normally. The
159

SwRI Project No. 14936.12

FIGURE 130. HORIBA PEMS VIBRATION POSITIONS

The Horiba PEMS experienced a mechanical failure during testing at the 75% energy level of the 

transverse horizontal vibration. A bracket holding a filter inside the Mechanical Enclosure (ME) 

box broke causing a flow line to become blocked. The test was stopped and the bracket was 

replaced with one from another unit. The testing was resumed until it was discovered that one of 

the pressure transducers used to measure the flow was not working properly causing an error in 

the reported flow. This problem was believed to have originated from the broken bracket because 

the line that was closed off was attached to this pressure transducer. The pressure transducer was 

replaced with one from the other unit and the system functioned properly. All future tests were 

conducted at 10%, 25%, and 50% energy for all PEMS to prevent further damage. The only 

other damage incurred by the Horiba was a rubber foot on the bottom of one of the boxes was 

sheared off. The measured dilution flow exhibited a higher degree of fluctuations during the 

vibration testing than was normally observed although the differences were not significant. 

The Sensors PEMS was operated without the exhaust flow meter and sample elbow attached to 

the unit. As can be seen in Figure 127, the two pieces were removed to allow a HEPA filter to be 

placed on the inlet. Although it would have been desirable to leave these pieces attached it was 

considered more important to provide clean air to ensure a zero particle level. The Sensors 
160

SwRI Project No. 14936.12

PEMS experienced no mechanical failures during vibration testing although several functionality 

issues did occur. The exhaust flow measurement became noisier with vibration although the 

magnitude of the noise was relatively small. In the EMI and RFI testing values as high as 1000 

kg/hr were observed, but in vibration testing the exhaust flow never went above 50 kg/hr. 

The Sensors PEMS experienced problems maintaining its total flow rate during vibration in the 

vertical direction. This problem was related to the automatic drain valves on the moisture traps in 

the dilution flow line. These traps are designed to open automatically when enough downwards 

force is applied from accumulated moisture in the reservoir, however the vertical vibration was 

causing the valves to repeatedly open allowing a portion of the flow to escape. Figure 131 shows 

an example of the total flow dropping while experiencing vertical vibration.

FIGURE 131. SENSORS TOTAL FLOW DURING VIBRATION TESTING

It was possible to test both of the AVL boxes simultaneously for the horizontal and vertical 

vibration, however they were tested separately for the 45 degree angle vibration. 

0

2

4

6

8

10

200 400 600 800 1000 1200 1400 1600

Total Flow (slpm)

Time (sec)

None 10% 25% 50% None
161

SwRI Project No. 14936.12

FIGURE 132. AVL PEMS VIBRATION POSITIONS

No mechanical problems were encountered with the AVL unit during vibration testing although 

the measurement did exhibit vibration induced noise. The measured soot concentration fluctuated 

with increasing amplitude as the vibration increased for all orientations tested. The peak 

measurement recorded was about plus and minus 0.2 mg/m3 with more typical values swinging 

between -0.1 mg/m3 and 0.1 mg/m3. A typical measurement is shown in Figure 133.
162

SwRI Project No. 14936.12

FIGURE 133. AVL SOOT MEASUREMENT NOISE DURING VIBRATION TESTING

This particular data was from the 45 degree angle longitudinal vibration. The noise in the 

measurement has no bias indicating that it will average out over larger periods of time. The 30 

second average is shown since this is likely the measurement error that would be observed 

during an NTE event. The maximum error of the 30 second average was -0.009 mg/m3 which is 

relatively insignificant based on PM concentrations expected at the threshold. However, it is 

unclear from this testing whether the noise could be greater if a particle source was present. The 

results from the vibration testing were presented to the steering committee at the September 22nd

6 MODELING RESULTS

, 

2009 meeting in Riverside. The steering committee declined to perform additional tests to create 

an error surface for vibration.

The main objective of this portion of the project was to use Monte Carlo techniques (e.g. random 

sampling) in an error model to simulate the combined effects of all the agreed-upon sources of 

PEMS error incremental to lab error on the components of the brake-specific (BS) PM 

emissions. This was accomplished by creating “error surfaces” for the Monte Carlo simulation 

to sample, based upon the results of a variety of lab experiments. The constructed model was 

simulated for thousands of trials (i.e., iterations) using data taken from a reference data set of 141

unique NTE events. The model results were used to determine the brake-specific additive 

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20

0 200 400 600 800 1000 1200

Reported Soot Concentration (mg/m3)

Time (sec)

Concentration dil. corrected [mg/m3] 30 Sec Avg

25% 50% 75% None
163

SwRI Project No. 14936.12

measurement allowances for PM by three different calculation methods for three different PEMS 

model units.

The error surfaces were generated from the results of each of the engine dynamometer and 

environmental chamber laboratory tests. The engine-lab-test error surfaces covered the domain 

of error versus the magnitude of the signal to which the error was to be applied (i.e., 1st to 99th

6.1 CONVERGENCE RESULTS FROM MC RUNS 

percentile error vs. concentration, flow, torque, etc.). The environmental-test error surfaces for 

shock and vibration, and electromagnetic and radio frequency interference (EMI/RFI) was not 

included because no error surfaces were generated. The environmental test error surfaces for 

pressure and temperature were characteristically different because they covered the domain of 

the environmental-test cycle time versus the magnitude of the signal to which the error was to be 

applied (i.e., error at a selected time vs. concentration). 

This section contains a summary of the checks to determine if the convergence criteria were met 

for the simulation runs. Section 2.9 on Convergence and Number of Trials contains a detailed 

description of the convergence methodology and the procedures followed to check for 

convergence for the reference NTE event trials. This procedure was applied to the simulation 

data obtained for each of the three PEMS units and all applicable calculation methods. 

Figure 134 through Figure 140 contain plots of the 90% confidence interval widths at the 95th

Table 22

percentile delta differences (expressed as a percent of the BSPM emissions NTE threshold) 

versus the ideal BSPM emissions for the 141 individual reference NTE events. This is done for 

each of the three PEMS units and the applicable calculation methods. A summary of the results 

is given in . Of interest was whether or not the simulations converged within 1% of the 

threshold value. As can be seen in the plots, a majority of the reference NTE events did not 

converge within 1% of the BSPM threshold. However, a majority did converge within 2% of the 

threshold. For the three PEMS units and calculation methods, the maximum percent of the 

confidence interval widths ranged from 1.82% for the AVL Method 1 to 2.76% for Sensors 

Method 1. Upon examination of the delta BSPM distributions for the various reference NTE 

events, those that had a high percentage above the BSPM threshold at the 95th percentile 

generally had low input PM concentration levels.
164

SwRI Project No. 14936.12

FIGURE 134. CONVERGENCE FOR AVL METHOD 1 AS A PERCENT OF BSPM 

THRESHOLD

0.00

0.40

0.80

1.20

1.60

2.00

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

AVL Method 1

0.00

0.40

0.80

1.20

1.60

2.00

2.40

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

AVL Method 2
165

SwRI Project No. 14936.12

FIGURE 135. CONVERGENCE FOR AVL METHOD 2 AS A PERCENT OF BSPM 

THRESHOLD

FIGURE 136. CONVERGENCE FOR AVL METHOD 3 AS A PERCENT OF BSPM 

THRESHOLD

0.00

0.40

0.80

1.20

1.60

2.00

2.40

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

AVL Method 3
166

SwRI Project No. 14936.12

FIGURE 137. CONVERGENCE FOR HORIBA METHOD 1 AS A PERCENT OF BSPM 

THRESHOLD

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Horiba Method 1

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Horiba Method 2
167

SwRI Project No. 14936.12

FIGURE 138. CONVERGENCE FOR HORIBA METHOD 2 AS A PERCENT OF BSPM 

THRESHOLD

FIGURE 139. CONVERGENCE FOR SENSORS METHOD 1 AS A PERCENT OF 

BSPM THRESHOLD

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

3.20

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Sensors Method 1
168

SwRI Project No. 14936.12

FIGURE 140. CONVERGENCE FOR SENSORS METHOD 2 AS A PERCENT OF 

BSPM THRESHOLD

TABLE 22. SUMMARY OF NUMBER OF REFERENCE NTES MEETING 2% 

CONVERGENCE

PEMS uNIT mETHOD mIN MAX # nTES WITHin 2% 

Convergence

% NTEs within 

2% 

CoNvergence

AVL 1 0.4467 1.8211 141 100%

2 0.4446 1.9766 141 100%

3 0.4240 1.9215 141 100%

Horiba 1 0.4158 2.5663 129 91%

2 0.4414 2.5447 131 93%

Sensors 1 0.7883 2.7664 125 87%

2 0.8613 2.6544 131 93%

6.2 SENSITIVITY BASED ON BIAS AND VARIANCE

This section contains a summary of the error surfaces that contributed the most to the bias 

of the generated BS emissions. The sensitivity charts developed in Crystal Ball help identify the 

error surfaces (assumptions) that are sensitive to changes in variation with respect to their effect 

on the three delta BS emissions. Another type of sensitivity examined in this study was 

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Sensors Method 2
169

SwRI Project No. 14936.12

concerned with the effects of potential “bias” in error surfaces and their effects on the forecast 

values. In order to study these effects a new error surface assumption was added to the MS 

simulation model for each of the original 31 error surfaces.

This assumption was sampled as a discrete binary distribution (i.e., on or off) during the 

simulation. For each trial of the simulation, 31 original error surfaces and 31 ‘on/off’ error 

surfaces were sampled according to their defined sample distribution. If the ‘on/off’ error 

surface produced an ‘off’ condition, the delta emissions from that particular error surface were 

not added to the BS emissions computations for the BS emissions ‘with errors’. Similarly, if the 

‘on/off’ error surface produced an ‘on’ condition, the delta emissions from that particular error 

surface were added to the BS emissions calculations.

During every trial of the simulation, the exclusions due to the ‘off’ conditions resulted in various 

combinations of the error surface delta emissions being added to the BS emissions ‘with errors’ 

computations. Over the course of a MC simulation with thousands of trials, the sensitivity of a 

particular error either ‘on’ or ‘off’ was assessed by examining the change in the forecast delta 

emission. Therefore, in a single MC simulation of a reference NTE event sensitivities due to 

variance and/or bias were explored.

Simulation results from the reference NTE events produced sensitivity values for all 95th 

percentile delta emissions by all three PEMS units and applicable calculation methods. Table 23

through Table 29 summarize the error surfaces in which either the contribution-to-variance 

normalized sensitivity value or the ‘on/off’ bias check for the error surface was at least 5% in 

magnitude compared to all the other error surfaces. If the label in the error surface contains the 

words ‘Delta’ then it represents a check for bias; otherwise, the error surface indicates a check 

for variance. Table 23 through Table 25 list the sensitivity and bias descriptive statistics for the 

delta BSPM emissions for the AVL PEMS for Methods 1, 2 and 3, respectively. For all three 

methods, the largest mean normalized variance was from the bias effect due to error surface #1, 

SSPM. 
170

SwRI Project No. 14936.12

TABLE 23. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR AVL BSPM METHOD 1

Error 

Surface 

# Error Surface

# Ref 

NTE 

Events

Avg Contribution 

to Normalized 

Variance,

%

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 141 11.34 7.59 35.25

2 TR PM 4 7.00 6.42 7.34

20 SS Exhaust 

Flow 4 5.57 5.32 6.05

31 Torque Warmup

10 -6.64 -11.22 -5.32

35 Torque Engine 

Manuf 1 -6.27 -6.27 -6.27

Delta Exhaust 

Flow Pulsation 3 5.78 5.56 5.99

Delta SS PM 141 -74.37 -83.67 -21.02

TABLE 24. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR AVL BSPM

Error 

Surface 

#

METHOD 2

Error Surface

# Ref 

NTE 

Events

Avg Contribution 

to Normalized 

Variance,

%

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 141 21.48 7.44 60.56

2 TR PM 38 9.20 5.53 12.31

31 Torque Warmup

8 -6.82 -10.77 -5.24

35 Torque Engine 

Manuf 1 -6.12 -6.12 -6.12

45 SS CO2 4 -6.89 -7.82 -5.32

Delta SS PM 141 -63.27 -83.77 -20.95
171

SwRI Project No. 14936.12

TABLE 25. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR AVL BSPM METHOD 3

Error 

Surface 

# Error Surface

# Ref 

NTE 

Events

Avg Contribution 

to Normalized 

Variance,

%

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 141 20.91 7.71 60.50

2 TR PM 36 9.27 5.17 12.52

31 Torque Warmup

9 -6.62 -10.77 -5.14

35 Torque Engine 

Manuf 1 -6.12 -6.12 -6.12

45 SS CO2 4 -6.96 -7.89 -5.40

Delta SS PM 141 -64.04 -83.76 -20.93

Table 26 through Table 27 list the sensitivity and bias descriptive statistics for the delta BSPM 

emissions for Methods 1 and 2, respectively. For both methods, the highest mean normalized 

variances were from the bias and variance due to error surface #1, SS PM. 

TABLE 26. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR HORIBA BSPM METHOD 1

Error 

Surface 

# Error Surface

# Ref 

NTE 

Events

Avg Contribution 

to Normalized 

Variance,

%

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 41.86 6.65 80.39

2 TR PM 10 5.84 5.04 7.18

20 SS Exhaust 

Flow 26 8.94 5.00 7.80

31 Torque Warmup

100 -8.34 -15.96 -5.08

35 Torque Engine 

Manuf 29 -5.72 -8.07 -5.00

Delta Exhaust 

Flow Pulsation 31 6.67 5.07 10.01

Delta SS PM 83 -32.05 -83.42 73.96
172

SwRI Project No. 14936.12

TABLE 27. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR HORIBA BSPM METHOD 2

Error 

Surface 

# Error Surface

# Ref 

NTE 

Events

Avg Contribution 

to Normalized 

Variance,

%

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 39.55 6.99 80.30

2 TR PM 10 5.89 5.00 7.42

31 Torque Warmup

95 -8.19 -15.04 -5.03

35 Torque Engine 

Manuf 24 -5.72 -7.65 -5.08

42 Fuel Rate 

Engine Manuf 7 5.52 5.07 6.34

45 SS CO2 54 -6.60 -9.83 -5.09

Delta SS PM 89 -42.60 -83.42 75.48

Table 28 through Table 29 list the sensitivity and bias descriptive statistics for the delta BSPM 

emissions for the Sensors PEMS for Methods 1 and 2, respectively. For both methods, the 

highest mean normalized variance was from the bias and variance due to error surface #1, SSPM.

TABLE 28. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR SENSORS BSPM METHOD 1

Error 

Surface 

# Error Surface

# Ref 

NTE 

Events

Avg Contribution 

to Normalized 

Variance,

%

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 46.92 6.57 78.61

2 TR PM 100 8.48 5.00 12.31

20 SS Exhaust 

Flow 8 6.11 8.06 7.56

31 Torque Warmup

30 -8.28 -11.57 -5.02

35 Torque Engine 

Manuf 10 -5.60 -6.06 -5.14

Delta Exhaust 

Flow Pulsation 9 7.01 5.56 8.79

Delta SS PM 120 -31.19 -86.58 75.35
173

SwRI Project No. 14936.12

TABLE 29. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141

REFERENCE NTE EVENTS FOR SENSORS BSPM METHOD 2

Error 

Surface 

# Error Surface

# Ref 

NTE 

Events

Avg Contribution 

to Normalized 

Variance,

%

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 45.92 6.68 77.78

2 TR PM 101 8.48 5.03 12.52

31 Torque Warmup

24 -7.84 -12.01 -5.08

35 Torque Engine 

Manuf 5 -5.61 -6.06 -5.01

42 Fuel Rate 

Engine Manuf 4 6.18 5.72 6.53

45 SS CO2 10 -6.84 -9.77 -5.15

Delta SS PM 127 -28.96 -86.67 75.75

The contribution to normalized variance and bias sensitivities from Table 23 through Table 29

are illustrated pictorially as box plots in Figure 141 through Figure 147 for BSPM by PEMS unit 

for Methods 1, 2 and 3. Only the error surfaces with at least 35 of the 141 reference NTE events 

(1/4 of the events) are included as box plots. The mean normalized variance for each of the 

plotted error surfaces is noted by a “+” symbol in the boxes. The error surface with the largest 

mean normalized variance is plotted at the left of the chart. The error surface with the second 

largest mean normalized variance is plotted second from the left, and so on. Figure 141 and 

Figure 142 demonstrate the high sensitivity to the negative bias for error surface #1, PM SS. 

Figure 145 and Figure 146 show a large variance effect due to PM SS.
174

SwRI Project No. 14936.12

FIGURE 141. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR AVL BSPM METHOD 1
175

SwRI Project No. 14936.12

FIGURE 142. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR AVL METHOD 2
176

SwRI Project No. 14936.12

FIGURE 143. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR AVL METHOD 3
177

SwRI Project No. 14936.12

FIGURE 144. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR HORIBA BSPM METHOD 1
178

SwRI Project No. 14936.12

FIGURE 145. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR HORIBA BSPM METHOD 2
179

SwRI Project No. 14936.12

FIGURE 146. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR SENSORS BSPM METHOD 1
180

SwRI Project No. 14936.12

FIGURE 147. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND 

VARIANCE FOR SENSORS BSPM METHOD 2
181

SwRI Project No. 14936.12

TABLE 30. SUMMARY OF ERROR SURFACE SENSITIVITIES TO BIAS AND 

VARIANCE FOR BSPM METHOD 1

Method 1

AVL Horiba Sensors

Error 

Surface 

# Error Surface

# NTE 

Events

Avg 

Contribution 

to 

Normalized 

Variance, %

# NTE 

Events

Avg 

Contribution 

to Normalized 

Variance, %

# NTE 

Events

Avg 

Contribution to 

Normalized 

Variance, %

1 SS PM 141 11.34 138 41.86 138 46.92

2 TR PM 4 7.00 10 5.84 100 8.48

20

SS Exhaust 

Flow 4 5.57 26 8.94 8 6.11

31

Torque 

Warm-up 10 -6.64 100 -8.34 30 -8.28

35

Torque 

Engine 

Manuf 1 -6.27 29 -5.72 10 -5.60

Delta 

Exhaust Flow 

Pulsation 3 5.78 31 6.67 9 7.01

Delta SS PM 141 -74.37 83 -32.05 120 -31.19
182

SwRI Project No. 14936.12

TABLE 31. SUMMARY OF ERROR SURFACE SENSITIVE TO BIAS AND 

VARIANCE FOR BSPM METHOD 2

Method 2

AVL Horiba Sensors

Error 

Surface 

# Error Surface

# NTE 

Events

Avg 

Contribution 

to 

Normalized 

Variance, %

# NTE 

Events

Avg 

Contribution 

to Normalized 

Variance, %

# NTE 

Events

Avg 

Contribution to 

Normalized 

Variance, %

1 SS PM 141 21.48 138 39.55 138 45.92

2 TR PM 38 9.20 10 5.89 101 8.48

31

Torque 

Warm-up 8 -6.82 95 -8.19 24 -7.84

35

Torque 

Engine 

Manuf 1 -6.12 24 -5.72 5 -5.61

42

Fuel Rate 

Engine 

Manuf 7 5.52 4 6.18

45 SS CO2 4 -6.89 54 -6.60 10 -6.84

Delta SS PM 141 -63.27 89 -42.60 127 -28.96
183

SwRI Project No. 14936.12

TABLE 32. SUMMARY OF ERROR SURFACE SENSITIVE TO BIAS AND 

VARIANCE FOR BSPM METHOD 3

Method 3

AVL

Error 

Surface 

# Error Surface

# NTE 

Events

Avg 

Contribution to 

Normalized 

Variance, %

1 SS PM 141 20.91

2 TR PM 36 9.27

31

Torque Warmup 9 -6.62

35

Torque Engine 

Manuf 1 -6.12

45 SS CO2 4 -6.96

Delta SS PM 141 -64.04

6.3 VALIDATION RESULTS

This section contains a summary of the model validation results, Section 2.13 on Validation

contains a more detailed description of the validation methodology utilized both in the 

simulation and in the on-road data collection efforts.

During the Monte Carlo simulation of the 141 reference NTE events some of the error surfaces 

were excluded in the computation of the BS emissions ‘with errors’ so that the simulation 

represented conditions used in collecting the on-road data. The error surfaces excluded were 

torque errors (# 29-32, 34, 35), fuel rate engine manufacturers (#42), dynamic speed (#43) and 

dynamic fuel rate (#44). For each reference NTE event, the difference in BSPM emissions was 

computed as

delta BSPM = BSPM with “Validation error” – “Ideal” BSPM.

These delta BSPM emissions were computed for each of the three PEMS units and all applicable 

calculation methods. The 5th, 50th and 95th percentiles were identified from the distributions of 
184

SwRI Project No. 14936.12

the delta BSPM emissions during the Monte Carlo simulation using the validation error surfaces 

only. Figure 148 through Figure 154 depict the validation percentiles for the AVL PEMS unit 

for methods 1, 2 and 3, respectively. Similar validation plots for the Horiba PEMS unit are 

illustrated in Figure 151 and Figure 152 for methods 1 and 2, respectively. Sensors PEMS 

validation plots for methods 1 and 2 are shown in Figure 153 and Figure 154.

FIGURE 148. VALIDATION PERCENTILES FOR THE 141 REFERENCE NTE 

EVENTS FOR AVL METHOD 1

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation AVL 1 5th Validation AVL 1 50th Validation AVL 1 95th
185

SwRI Project No. 14936.12

FIGURE 149. VALIDATION PERCENTILES FOR THE 141 REFERENCE NTE 

EVENTS FOR AVL METHOD 2

FIGURE 150. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR AVL METHOD 3

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation AVL 2 5th Validation AVL 2 50th Validation AVL 2 95th

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation AVL 3 5th Validation AVL 3 50th Validation AVL 3 95th
186

SwRI Project No. 14936.12

FIGURE 151. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR HORIBA METHOD 1

FIGURE 152. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR HORIBA METHOD 2

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.020

0.025

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Horiba 1 5th Validation Horiba 1 50th Validation Horiba 1 95th

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.020

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Horiba 2 5th Validation Horiba 2 50th Validation Horiba 2 95th
187

SwRI Project No. 14936.12

FIGURE 153. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR SENSORS METHOD 1

FIGURE 154. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR SENSORS METHOD 2

-0.035

-0.030

-0.025

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Sensors 1 5th Validation Sensors 1 50th Validation Sensors 1 95th

-0.035

-0.030

-0.025

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Sensors 2 5th Validation Sensors 2 50th Validation Sensors 2 95th
188

SwRI Project No. 14936.12

The 5th and 95th percentiles of the validation delta BSPM were separately fit for the AVL and 

Sensors PEMS units using a simple linear regression model. However, the criteria for accepting 

the linear fits was not met for any of the calculations methods for these two PEMS units. Thus, 

loess regression fits were used to determine the best functional representation for the 5th and 95th

delta BSPM based on the validation simulation modeling. These loess fits for the 95th and 5th

percentiles for the Sensors unit methods 1 and 2 can be found in Figure 155 through Figure 158, 

respectively. The loess fits for the 95th and 5th

Figure 159

percentiles for the AVL units methods 1, 2 and 3 

can be found in through Figure 162, respectively. The loess smoothing parameters 

for the regression fits are listed in Table 33.

TABLE 33. LOESS SMOOTHING PARAMETERS FOR VALIDATION PERCENTILES

PEMS Method 5th Percentile 95th

Sensors

Percentile

1 0.290 0.570

2 0.290 0.570

AVL 1 0.294 0.755

2 0.294 0.777

3 0.294 0.777
189

SwRI Project No. 14936.12

FIGURE 155. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

SENSORS METHOD 1
190

SwRI Project No. 14936.12

FIGURE 156. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

SENSORS METHOD 1

FIGURE 157. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR

SENSORS METHOD 2
191

SwRI Project No. 14936.12

FIGURE 158. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

SENSORS METHOD 2

FIGURE 159. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 1
192

SwRI Project No. 14936.12

FIGURE 160. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 1

FIGURE 161. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 2
193

SwRI Project No. 14936.12

FIGURE 162. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 2

FIGURE 163. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 3
194

SwRI Project No. 14936.12

FIGURE 164. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 3

6.4 MEASUREMENT ERROR ALLOWANCE RESULTS 

This section contains a summary of the measurement error allowance results using both a 

regression method and a median method to determine the measurement allowance. Section 2.12

on Measurement Allowance contains a detailed description of the methodology followed in 

determining these values. This procedure was applied to the simulation data for all 141 reference 

NTE events obtained for all three calculation methods for the AVL PEMS and for calculations 

methods 1 and 2 for the Horiba and Sensors PEMS. 

Figure 165 contains a regression plot of the 95th percentile delta BSPM values versus the Ideal 

BSPM values for the 141 reference NTE events for AVL Method 1. Included in the plot is the 

equation for the fitted regression line, and the R-square (R2

) value and root mean square error 

(RMSE) value for the regression fit. The two symbols in the plot represent reference NTE 

events where there was a dominant bias effect due to the SSPM error surface (diamond symbol) 

or there was a dominant variance effect due to the SSPM error surface (square symbol). The Rsquare value indicates that 47.95% of the variation in the 95th percentile BSPM values is 

explained by the ideal BSPM values for the AVL Method 1 data. The RMSE value of 0.0008 

displays the size of the estimated standard deviation of the predicted 95th percentile BSPM 

values.
195

SwRI Project No. 14936.12

Table 34 includes a comparison of the results of the regression method based on Figure 165 and 

the median method as described in the Section 2.10 on Measurement Allowance. Under the 

heading of “Regression Method” in the table, it is shown that only the R-square criterion was not 

met by the data. Thus, the Median Method must be used. Under the heading “Median Method” in 

the table, the measurement error at the BSPM threshold, based on using the median of the 141 

95th percentile delta BSPM values, is 0.661% when expressed as a percent of the threshold of 

0.02 g/hp-hr.

FIGURE 165. REGRESSION PLOT OF 95TH

y = 0.116x - 0.002

R2 = 0.4795

RMSE = 0.0008

-0.0020

-0.0010

0.0000

0.0010

0.0020

0.0030

0.0040

0.0050

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM g/hp-hr

Ideal BSPM, g/hp-hr

AVL Method 1

Delta SSPM ic SSPM

PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR AVL METHOD 1
196

SwRI Project No. 14936.12

TABLE 34. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR AVL METHOD 1

Regression Method Median Method

R 0.4795 2 Did Not Meet 

Criteria

RMSE (SEE) 0.0008 Met Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0003399 % Delta at 

Threshold Median 95th % Delta 0.0001322

Measurement Error @ 

Threshold = 0.02 1.6993% Measurement Error @ Threshold = 0.02 0.661%

Figure 166 contains a regression plot of the 95th percentile delta BSPM values versus the Ideal 

BSPM values for the 141 reference NTE events for AVL Method 2. The R-square value 

indicates that 46.02% of the variation in the 95th

Table 35

percentile BSPM values is explained by the 

Ideal BSPM values for the AVL Method 2 data. The RMSE value is 0.0008.

includes a comparison of the results of the regression method based on Figure 166 and 

the median method. Under the heading of “Regression Method” in the table, it is shown that the 

R-square criterion for using this method is not met by the data. Thus, the Median Method must 

be used. Under the heading “Median Method” in the table, the measurement error at the BSPM 

threshold, based on using the median of the 141 95th percentile delta BSPM values, is -2.375% 

when expressed as a percent of the threshold value of 0.02.
197

SwRI Project No. 14936.12

FIGURE 166. REGRESSION PLOT OF 95TH

TABLE 35. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR AVL METHOD 2

PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR AVL METHOD 2

Regression Method Median Method

R 0.4602 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0008 Met Criteria

5% Median Ideal 0.0191007

Predicted 95th

-0.0002124 % Delta at 

Threshold

Median 95th

-0.0004751 % 

Delta

Measurement Error @ 

Threshold = 0.02 -1.0618% Measurement Error @ Threshold = 0.02 -2.375%

Figure 167 contains a regression plot of the 95th percentile delta BSPM values versus the Ideal 

BSPM values for the 141 reference NTE events for AVL Method 3. The R-square value 

indicates that 46.20% of the variation in the 95th

y = 0.0996x - 0.0022

R2 = 0.4602

RMSE = 0.0008

-0.0020

-0.0010

0.0000

0.0010

0.0020

0.0030

0.0040

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

AVL Method 2

Delta SSPM ic SSPM

percentile BSPM values is explained by the 

Ideal BSPM values for the AVL Method 3 data. The RMSE value is 0.0008.
198

SwRI Project No. 14936.12

Table 36 includes a comparison of the results of the regression method based on Figure 167 and 

the median method. Under the heading of “Regression Method” in the table, it is shown that the 

R-square criterion for using this method is not met by the data. Thus, the Median Method must 

be used. Under the heading “Median Method” in the table, the measurement error at the BSPM 

threshold, based on using the median of the 141 95th percentile delta BSPM values, is -2.383% 

when expressed as a percent of the threshold value of 0.02.

FIGURE 167. REGRESSION PLOT OF 95TH

y = 0.1001x - 0.0022

R2 = 0.462

RMSE = 0.0008

-0.0020

-0.0010

0.0000

0.0010

0.0020

0.0030

0.0040

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

AVL Method 3

Delta SSPM ic SSPM

PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR AVL METHOD 3
199

SwRI Project No. 14936.12

TABLE 36. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR AVL METHOD 3

Regression Method Median Method

R 0.4620 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0008 Met Criteria

5% Median Ideal 0.0191007

Predicted 95th

-0.0002118 % Delta at 

Threshold

Median 95th

-0.0004766 % 

Delta

Measurement Error @ 

Threshold = 0.02 -1.0592% Measurement Error @ Threshold = 0.02 -2.383%

Figure 168 contains a regression plot of the 95th percentile delta BSPM values versus the Ideal 

BSPM values for the 141 reference NTE events for Horiba Method 1. The R-square value 

indicates that 26.57% of the variation in the 95th

Table 37

percentile BSPM values is explained by the 

Ideal BSPM values for the Horiba Method 1 data. The RMSE value is 0.0041.

includes a comparison of the results of the regression method based on Figure 168 and 

the median method. Under the heading of “Regression Method” in the table, it is shown that the 

R-square and the RMSE criteria for using this method were not met by the data. Thus, the 

Median Method must be used. Under the heading “Median Method” in the table, the 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 54.379 % when expressed as a percent of the threshold value of 0.02.
200

SwRI Project No. 14936.12

FIGURE 168. REGRESSION PLOT OF 95TH

TABLE 37. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR HORIBA METHOD 1

PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR HORIBA METHOD 1

Regression Method Median Method

R 0.2657 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0041 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0102566 % Delta at 

Threshold

Median 95th

0.0108759 % 

Delta

Measurement Error @ 

Threshold = 0.02 51.2831% Measurement Error @ Threshold = 0.02 54.379%

Figure 169 contains a regression plot of the 95th percentile delta BSPM values versus the Ideal 

BSPM values for the 141 reference NTE events for Horiba Method 2. The R-square value 

indicates that 25.97 % of the variation in the 95th

y = -0.3555x + 0.0174

R2 = 0.2657

RMSE = 0.0041

-0.0050

0.0000

0.0050

0.0100

0.0150

0.0200

0.0250

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Horiba Method 1

Delta SSPM ic SSPM

percentile BSPM values is explained by the 

Ideal BSPM values for the Horiba Method 2 data. The RMSE value is 0.0041.
201

SwRI Project No. 14936.12

Table 38 includes a comparison of the results of the regression method based on Figure 169 and

the median method. Under the heading of “Regression Method” in the table, it is shown that the 

R-square and the RMSE criteria for using this method were not met by the data. Thus, the 

Median Method must be used. Under the heading “Median Method” in the table, the 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 50.079 % when expressed as a percent of the threshold value of 0.02.

FIGURE 169. REGRESSION PLOT OF 95TH

y = -0.3515x + 0.0165

R2 = 0.2597

RMSE = 0.0041

-0.0050

0.0000

0.0050

0.0100

0.0150

0.0200

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Horiba Method 2

Delta SSPM ic SSPM

PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR HORIBA METHOD 2
202

SwRI Project No. 14936.12

TABLE 38. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR HORIBA METHOD 2

Regression Method Median Method

R 0.2597 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0041 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0094282 % Delta at 

Threshold

Median 95th

0.0100158 % 

Delta

Measurement Error @ 

Threshold = 0.02 47.1408% Measurement Error @ Threshold = 0.02 50.079%

Figure 170 contains a regression plot of the 95th percentile delta BSPM values versus the Ideal 

BSPM values for the 141 reference NTE events for Sensors Method 1. The R-square value 

indicates that 45.49 % of the variation in the 95th

Table 39

percentile BSPM values is explained by the 

Ideal BSPM values for the Sensors Method 1 data. The RMSE value is 0.0029.

includes a comparison of the results of the regression method based on Figure 170 and

the median method. Under the heading of “Regression Method” in the table, it is shown that the 

R-square and the RMSE criteria for using this method were not met by the data. Thus, the 

Median Method must be used. Under the heading “Median Method” in the table, the 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 34.361 % when expressed as a percent of the threshold value of 0.02.
203

SwRI Project No. 14936.12

FIGURE 170. REGRESSION PLOT OF 95TH

TABLE 39. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR SENSORS METHOD 1

PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR SENSORS METHOD 1

Regression Method Median Method

R 0.4549 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0029 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0066785 % Delta at 

Threshold

Median 95th

0.00687227 % 

Delta

Measurement Error @ 

Threshold = 0.02 33.3924% Measurement Error @ Threshold = 0.02 34.361%

Figure 171 contains a regression plot of the 95th percentile delta BSPM values versus the Ideal 

BSPM values for the 141 reference NTE events for Sensors Method 2. The R-square value 

indicates that 45.03 % of the variation in the 95th

y = -0.3866x + 0.0144

R2 = 0.4549

RMSE = 0.0029

-0.0150

-0.0100

-0.0050

0.0000

0.0050

0.0100

0.0150

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Sensors Method 1

Delta SSPM ic SSPM

percentile BSPM values is explained by the 

Ideal BSPM values for the Sensors Method 1 data. The RMSE value is 0.0030.
204

SwRI Project No. 14936.12

Table 40 includes a comparison of the results of the regression method based on Figure 171 and 

the median method. Under the heading of “Regression Method” in the table, it is shown that the 

R-square and the RMSE criteria for using this method were not met by the data. Thus, the 

Median Method must be used. Under the heading “Median Method” in the table, the 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 30.285 % when expressed as a percent of the threshold value of 0.02.

FIGURE 171. REGRESSION PLOT OF 95TH

y = -0.3875x + 0.0137

R2 = 0.4503

RMSE = 0.0030

-0.0150

-0.0100

-0.0050

0.0000

0.0050

0.0100

0.0150

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Sensors Method 2

Delta SSPM ic SSPM

PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR SENSORS METHOD 2
205

SwRI Project No. 14936.12

TABLE 40. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR SENSORS METHOD 2

Regression Method Median Method

R 0.4503 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0030 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0059333 % Delta at 

Threshold

Median 95th

0.0060569 % 

Delta

Measurement Error @ 

Threshold = 0.02 29.6663% Measurement Error @ Threshold = 0.02 30.285%

Table 41 contains a summary of the measurement error values contained in Table 34 through 

Table 40. The values are categorized by PEMS unit and by calculation method.

TABLE 41. BSPM MEASUREMENT ERROR IN PERCENT OF NTE THRESHOLD BY 

PEMS AND CALCULATION METHOD

Measurement Errors (%) at Respective NTE Threshold

PEMS

Method 1

Exhaust Flow 

Torque-Speed

Method 2

Exhaust and Fuel 

Flow TorqueSpeed

Method 3

Fuel Flow Torque-Speed

AVL 0.661 -2.375 -2.383

Horiba 54.379 50.079 n/a

Sensors 34.361 30.285 n/a

Table 42 includes in the measurement allowance selected based on the minimum normalized 

PM. The AVL was not used in the measurement allowance determination because the AVL at 

the start of the program was not accepted as an official PEMS, and the measurement Steering 

Committee had decided that the measurement allowance would only be based on the Sensors or 

the Horiba PEMS.
206

SwRI Project No. 14936.12

TABLE 42. MEASUREMENT ALLOWANCE AT NTE THRESHOLD BY EMISSIONS 

FOR METHOD 2

PEMS

Method 2 

Measurement 

Error %

NTE Threshold 

g/hp-hr

Measurement 

Allowance, 

g/hp-hr

Sensors 30.285 0.02 0.00605

On-road PM emissions were gathered from selected routes driven to collect emissions data with 

a CE-CERT trailer and a PEMS installed on the tractor pulling the trailing. For each on-road 

NTE event, a delta BSPM emissions value was computed as follows:

Delta BSPM = PEMS BSPM – CE-CERT BSPM.

These differences were computed for the BSPM emissions for each PEMS unit tested and their 

corresponding calculation methods. The on-road delta BSPM emissions were calculated for the 

The majority of CE-CERT validation data were produced without any diesel particle filter (DPF) 

active regeneration “no regen”. Some data were produced during DPF active regeneration 

“regen” and those were used for information only, as the lab validation data did not include DPF 

active regeneration events. AVL methods 1, 2 and 3 including three individual units (#2, #3 and 

#4) and 271 NTE events. This data set was computed as “no regen”. In addition, AVL units #2, 

#3 and #4 collected 21 NTE event BSPM differences under the “regen” scenario. These “regen” 

data were not used in the validation decision but are presented for information purposes only. 

The second PEMS unit tested for validation was the Sensors. The on-road delta BSPM 

emissions were calculated for Sensors methods 1 and 2 also using three individual units in the 

“no regen” scenario and resulted in 217 NTE events.

The loess regression fits in Figure 172 through Figure 175 and the Sensors CE-CERT BSPM 

differences collected on-road were plotted in order to determine if the simulation method 

validated. If the number of CE-CERT delta BSPM values does not exceed 10% of the total 

number of on-road NTE events collected, then the simulation method would be considered 

valid. Figure 172 represents the validation plot for the BSPM method 1 analysis for Sensors. 

Note that 7 of the 217 on-road NTE events were either below or above the range of the ideal 

BSPM and were excluded from the validation percentage calculation. Therefore, 68 of the 210 

CE-CERT NTE events (32.38%) fell below the simulation model 5th percentile based on the 

loess regression. Thus, the model was not considered valid for the BSPM Sensors Method 1.
207

SwRI Project No. 14936.12

FIGURE 172. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM SENSORS METHOD 1 WITH NO REGEN

Figure 173 represents the validation plot for the BSPM method 2 analysis for Sensors. Note that 

again 7 of the 217 on-road NTE events were either below or above the range of the ideal BSPM 

and were excluded from the validation percentage calculation. Therefore, 71 of the 210 CECERT NTE events (33.81%) fell above the simulation model 95th percentile or below the 

simulation model 5th percentile based on the loess regressions. Thus, the model was not 

considered valid for the BSPM Sensors Method 2.
208

SwRI Project No. 14936.12

FIGURE 173. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM SENSORS METHOD 2 WITH NO REGEN

Figure 174 represents the validation plot for the BSPM method 1 analysis for AVL with ‘no 

regen’. In this case 8 of the 271 on-road NTE events were either below or above the range of 

the ideal BSPM and were excluded from the validation percentage calculation. Therefore, 49 of 

the 263 CE-CERT NTE events (18.63%) fell above the simulation model 95th percentile or 

below the simulation model 5th percentile based on the loess regressions. Thus, the model was 

not considered valid for the BSPM AVL Method 1.
209

SwRI Project No. 14936.12

FIGURE 174. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 1 WITH NO REGEN

Figure 175 represents the validation plot for the BSPM method 2 analysis for AVL with ‘no 

regen’. Again in this case 8 of the 271 on-road NTE events were either below or above the 

range of the ideal BSPM and were excluded from the validation percentage calculation. 

Therefore, 30 of the 263 CE-CERT NTE events (11.41%) fell above the simulation model 95th

percentile or below the simulation model 5th percentile based on the loess regressions. Thus, the 

model was not considered valid for the BSPM AVL Method 2.
210

SwRI Project No. 14936.12

FIGURE 175. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 2 WITH NO REGEN

Figure 176 represents the validation plot for the BSPM method 3 analysis for AVL with ‘no 

regen’. Again in this case 8 of the 271 on-road NTE events were either below or above the 

range of the ideal BSPM and were excluded from the validation percentage calculation. 

Therefore, 26 of the 263 CE-CERT NTE events (9.89%) fell above the simulation model 95th

percentile or below the simulation model 5th percentile based on the loess regressions. Thus, the 

model was considered valid for the BSPM AVL Method 3 since the number of CE-CERT NTE 

events outside the 5th and 95th percentile loess regression was less than 10%.
211

SwRI Project No. 14936.12

FIGURE 176. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 3 WITH NO REGEN

Figure 177 through Figure 179 illustrate the validation plots for the AVL methods 1, 2, and 3, 

respectively, based on the ‘regen’ computation. These were not included in the validation 

analysis but are provided for information purposes only.
212

SwRI Project No. 14936.12

FIGURE 177. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 1 WITH REGEN
213

SwRI Project No. 14936.12

FIGURE 178. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 2 WITH REGEN
214

SwRI Project No. 14936.12

FIGURE 179. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 3 WITH REGEN

TABLE 43. SUMMARY OF BSPM MODEL VALIDATION RESULTS

PEMS Unit

Method 1

Exhaust Flow TorqueSpeed

Method 2

Exhaust and Fuel 

Flow Torque-Speed

Method 3

Fuel Flow TorqueSpeed

Sensors “no regen” No No No

AVL “no regen” No No Yes
215

SwRI Project No. 14936.12

7 SUMMARY

A series of engine experiments and environmental tests were performed on three PM-PEMS that 

included the Sensors PPMD, Horiba TRPM, and AVL MSS. The Sensors PPMD and Horiba 

TRPM were treated as official PEMS, and only those were used for official results and 

determination of next steps. The AVL MSS were used in conjunction with the Sensors PPMD 

based on an agreement reached between Sensors and AVL. Error surfaces were developed based 

on experimental work to determine PM-PEMS bias and precision errors using Monte Carlo 

simulation model. The output of the model was to determine the error distribution at a set of 

reference NTE events, compared to their ideal value. The PM-PEMS with the lowest 95th

percentile error that is greater than zero was selected for in-use validation testing of the model.

The PM-PEMS that was selected for in-use validation was the Sensors PPMD. The PPMD 

produced a 95th percentile measurement allowance error of 0.006 g/hp-hr at a threshold NTE 

limit of 0.02 g/hp-hr using Method 2, compared to the 0.01 g/hp-hr that was produced by the 

Horiba TRPM. As for the AVL MSS, the instrument produced zero measurement allowance, but 

its measurement allowance value was not officially used because the MSS only measures the 

carbon fraction of PM, compared to the required total (solid plus volatile) PM measured by the 

other two PM-PEMS. However, because the AVL MSS was used in conjunction with the PPMD, 

the SC agreed to include it during in-use validation testing. Due to funding limitation, the 

Horiba TRPM was not included in in-use validation testing.

Based on in-use validation testing, the Sensors PPMD failed validation because 32 percent and 

34 percent of the data produced in-use were below the 5th percentile of the validation window, 

using Method 1 and Method 2, respectively. The SC agreed during the development of the Test 

Plan that less than 10 percent of the data need to be outside the validation window to pass 

validation. 

As for the AVL MSS, it passed validation using Method 3 by having 10 percent of the data 

outside the validation window, with the majority of the data being higher the 95th percentile. 

Method 2 failed by two percentage points and Method 1 failed by 8 percentage points.

Because the MSS Method 3 passed validation based on its own validation window, the SC 

concluded the measurement allowance program, and accepted the measurement allowance based 

on the Sensors PPMD as the final PM-PEMS measurement allowance. 

8 REFERENCES

1. “Determination of PEMS Measurement Allowance for Gaseous Emissions Regulated Under 

the Heavy-Duty In-Use Testing Program: Revised Final Report,” EPA420-R-08-005, 

February 2008.
216

SwRI Project No. 14936.12

2. “Electronic Code of Federal Regulations, Title 40 Protection of the Environment, Vol. 15 

Part 1065,” http://www.ecfr.gpoaccess.gov, 2009.

3. Conover, W.J., Practical Nonparametric Statistics, John Wiley & Sons, 1971

4. Cleveland, W.S. , “Robust Locally-Weighted Regression and Smoothing Scatterplots.” 

Journal of the American Statistical Association, 1979, 74, pp. 829-836.

5. Hurviuch, C.M., Simonoff, J.S., and Tsai, C.L., “Smoothing Parameters Selection in 

Nonparametric Regression Using and Improved Akaike Information Criterion,” Journal of 

the Royal Statistical Society B, 1998, 60, pp.271-293.

6. Stratmann, F., Otto, E., and Fissan, H., “Thermophoretical and Diffusional Particle Transport 

in a Cooled Laminar Tube Flow.” Journal of Aerosol Science, March 28th, 1994, vol. 25; pp. 

1359-1365.
217

SwRI Project No. 14936.12

APPENDIX A: PEMS OPERATION LOG
218

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

2/25/2008 Horiba

Lost communication with 

TRPM laptop on a 

regular basis

Standard ethernet 

cable does not fit 

properly into this 

laptop

A industrial grade ethernet 

cable was provided by 

Horiba which solved this 

problem

2/28/2008 Horiba 1

Could not control dilution 

ratio if external flow 

meter is not connected

Unknown Horiba fixed the problem

2/28/2008 Horiba 2

Java software does not 

display the measured 

values that are in the 

Labview software

Unknown Software update fixed the 

problem

3/3/2008 Horiba 2 Dilution air flow not 

stable

Dilution air 

pressure too high

Performed a dilution air 

flow adjustment per 

Horiba

3/4/2008 Sensors 1 Lookup table for MPS2 

repeatedly failed Unknown

Sensors said the criteria is 

too stringent and it is fine 

as long as it visually looks 

good

3/4/2008 Horiba 1 Dilution Ratio control is 

still somewhat erratic

Bad PID 

constants

Adjusted PID constants 

to new values suggested 

by Horiba

3/14/2008 Horiba 1

Pressure transducer Pt1 

would not respond to 

calibration

Pressure 

transducer was 

broken

Replaced with new part 

from Horiba

3/21/2008 Horiba 

1,2,&3

Unable to pass Part 1065 

sample flow linearity 

verification

Sample flow is an 

inferred not 

measured value

Linearity verification 

performed on dilution and 

total flow at MASC 

request

3/24/2008 Horiba 

1,2,&3

Sample flow failed 

Horiba check with 

provided external flow 

meter

Flow calibrations 

needed updating

Flow coefficients updated 

for all three units

3/26/2008 Horiba 

1,2,&3

Unable to pass Part 1065 

dilution flow linearity 

verification

Dilution flow 

coefficients 

needed updating

Performed "Dilution Ratio 

Accuracy Adjustment" as 

instructed by Horiba
219

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

4/14/2008 Horiba 3 Java software freezes 

when loading

Wrong 

parameters set in 

a config file

AVL programmed an 

offset into their software 

to account for this, laterit 

appeared to be a 

grounding problem in the 

Semtech DS

4/22/2008 Horiba 3

Java softare is not 

communicating with the 

Labview software

Improper 

software 

configuation

Parameters adjusted to fix 

the problem

6/13/2008 Horiba 

2200

Horiba system is unable 

to log the ISO-1576 

ECM broadcast from the 

International Engine

Software did not 

have this 

capability

International engine was 

replaced with a heavyduty Volvo engine that 

uses J1939 broadcast

6/19/2008 Horiba 1

TRPM software not 

reading the same exhaust 

flow as OBS-2200 

software

Calibration 

coefficient is 

wrong

Manually adjust the 

calibration coefficient to 

get the readings to match

6/28/2008 Sensors 1

Sample valve for crystal 

remains in "transient" 

state every time it 

attempts to sample

Stepper motor 

attempting to turn 

too quickly (not 

enough torque)

Sensors readjusted the 

stepper motor speed

7/31/2008 Horiba 2

The OBS-2200 software 

would not trigger the 

OBS-TRPM to start 

sampling during an NTE 

Connector wired 

incorrectly

Reduced the speed of the 

stepper motor

8/8/2008 Horiba 2 Dilution flow is too low Replaced with connector 

from second unit

8/12/2008 Horiba 

2200

OBS-2200 software 

unable to read the 

reference torque value 

from J1939 broadcast

Unable to enter 

the proper data 

bit location

Perform dilution flow 

adjustment using internal 

pressure regulator

8/14/2008 Sensors 2 Could not get any of the 

crystals to os cillate

Power supply in 

CQCM head was 

likely burnt out

Enter the reference torque 

value manually

8/15/2008 Horiba 

2200

OBS-2200 laptop would 

not boot up Unknown

Hard drive was placed in 

another identical laptop, 

broken laptop was 

shipped back to Horiba
220

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

8/18/2008 Sensors 1 Bypass flow was not 

updating

8/21/2008 Sensors 1

Crystal frequencies, 

corona currents, and 

voltages were dropping 

out

High temperature 

causing 

communication 

problems

Attempted to reduce test 

cell temperature, but 

problem persisted

8/26/2008 AVL 1

Unit switched into Zero 

Check in the middle of 

the test

Uknown, possibly 

a pressure out of 

limit

Unit switched back into 

sample after approx. 30 

seconds

8/27/2008 Horiba 1 OBS-2200 laptop failed, 

would no longer boot Laptop broken New laptop provided by Horiba

8/27/2008 Horiba 1 OBS-2200 laptop would 

not read J1939 Unknown Sent back to Horiba for repair

8/27/2008 Horiba 1 TPA would not zero

Zero function tied 

to gas analyzers, 

looking for gas 

flow

Use the CAL function 

instead of the ZERO to 

just zero TPA

8/27/2008 Horiba 1

TRPM Java software 

unable to log exhaust 

flow (Labview works)

Error in software Installed new version of 

Java software

8/27/2008 Horiba 1 EAD check will fail 

repeatedly

Tolerances are 

too tight for test 

cell operation, 

tolerances 

relaxed.

Ignored tolerances set in 

software for check

8/27/2008 AVL 1

Analog output signal to 

MSS would clip at 2 

mg/m3 range

Soot 

concentration too 

high

Output switched to 0-10 

mg/m3, DR setpoint 

increased from 3 to 6

8/27/2008 AVL 1 DR would oc casionally 

stop controlling

Too much 

moisture in the 

system

Unit purged for moisture 

overnight, and firmware 

upgraded

8/27/2008 Sensors 1

PPMD internal 

temperature was out of 

limit

Test cell 

temperature too 

hot

A new back panel was 

installed with two fans to 

promote cooling

8/27/2008 Sensors 1

Bypass flow from TSI 

flowmeter was not 

reading in software

Faulty com cable Replaced TSI cable
221

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

8/27/2008 Sensors 1

CQCM communication 

dropped out during 

testing

CQCM power 

supply failed Power supply replaced

8/27/2008 Sensors 1

CQCM communication 

dropped out during 

testing

Unknown

CQCM comm chip 

replaced with a newer 

model

8/27/2008 Sensors 1 Corona needle high 

voltage erratic Unknown Fixing the comm issue resolved this problem

8/27/2008 Sensors 1 PPMD communication 

was dropping out

High internal 

temperature

Directed cooling air at 

PPMD

9/10/2008 Horiba 1

Compressor tripped 

breaker, turned off AVL 

unit

Combining AVL 

and compressor 

on same 15A 

circuit was too 

much current

Horiba compressor 

moved to dedicated 15A 

circuit

9/10/2008 Horiba 1

Make up air flow was 

0.7 lpm instead of 2.3 

lpm

PID constants 

incorrect Modified PID constants

9/12/2008 AVL 1

Analog output signal to 

MSS would still briefly 

clip at 10 mg/m3 range

Soot 

concentration too 

high

Logarithmic analog output 

added as a firmware 

update

9/12/2008 Sensors 1

Negative emissions 

reported even at high 

emission levels on NTE

High dilution, 

crystal saturation 

(no grease), 

crystal 

stabilization

Do not use MPS2, grease 

crystals, wait longer for 

PPMD to warm up

9/18/2008 Sensors 1

The bypass flow would 

increase when crystal 1 

on PPMD1 would 

sample

Crystal may be 

installed 

backwards so 

that it is always 

sampling

Set crystal 1 to be the 

reference crystal

9/18/2008 Sensors 1

Sample flow drifting 

during steady state engine 

operation

Temperature 

estimate is based 

on mixing of two 

flows, if the 

estimate is off 

temperature will 

change the flow

Adjusted the parameters 

in the temperature 

estimate
222

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

9/18/2008 Sensors 1 No corona current was 

measured for two crystals

Crystals were 

shorted to ground Crystals were replaced

9/18/2008 Sensors 1 High voltage reading was 

low on two crystals

Corona needles 

were too close to 

the crystals

Needles repositioned

9/20/2008 Sensors 1

Sample flow 

measurement was too 

low

Sample flow TC 

was in excess 

flow return, which 

had low flow due 

to connection to 

CVS

TC moved into the main 

exhaust

9/26/2008 AVL 1 Error: MFD Temperature 

out of Spec

Test cell 

temperature too 

hot

Repos itioned chiller air to 

blow on the AVL unit

9/30/2008 Sensors 1 Lower than expected 

emissions for the PPMD

Inaccurate sample 

flow temp causing 

low sample flow 

meas, corona 

needles not 

positioned 

properly

Repos ition the sample 

flow thermocouple, adjust 

corona needle

10/9/2008 Sensors 1

PPMD sampling delayed 

several seconds after 

trigger

Delay in 

communications

Increased residcence time 

inside PPMD up to 3 

seconds

10/9/2008 Sensors 1 Inaccuracies in flow 

measurement

System uses an 

assumed inlet 

pressure for 

MPS2

Added pressure 

measurement downstream 

of MPS1 to account for 

changes in pressure due 

to MPS2

10/9/2008 Sensors 1

Crystal sampling as soon 

as it becomes available, 

including in an NTE event

Software logic

Software modified so that 

an available crystal waits 

for the start of the next 

NTE event

10/9/2008 Horiba 1

Compressor supplying 

dilution air stops working 

in middle of test (2 units)

Unit shutting off 

due to overheat 

protection

Cool air provided to the 

compressor (temporary 

solution)
223

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

10/9/2008 AVL 1

MSS concentration 

reporting too high in 

Sensors software

Sensors not 

correcting 

concentration 

from 0°C to 

20°C (~8%)

Semtech DS software 

updated to include the log 

range and volume 

correction

10/10/2008 Sensors 1 Bypass flow too low

Flow leaking 

through the 

carbon filter 

connection

Tightened the carbon filter

10/15/2008 Sensors 1 PPMD emission results 

were inconsistent

Booker had 

decreased the 

crystal flow from 

0.4 to 0.2 slpm to 

increase loading 

time

Final crystal sample flow 

set at 0.5 slpm

10/15/2008 Sensors 1 PPMD emissions were 

lower than expected Uknown

Crystal sensitivity 

adjusted from 125 hz/μg 

to 100 h z/μg (increasing 

sensitivity by 25%)

10/15/2008 Sensors 1

PPMD wouldn't sample 

when external trigger 

activates on Semtech DS

Unknown

Powered off hardware 

and laptop and restarted, 

problem was resolved

10/16/2008 Sensors 1 PPMD unable to 

communicate with DS Unknown

Triggered the PPMD 

manually through Host 

software (unofficial 

testing)

10/28/2008 Horiba 1 Dilution flow inaccurate Unknown Recalibrated VFM

11/3/2008 Sensors 1 PPMD block pressure 

low

Uknown, block 

pressures were 

activated in the 

software

Aborted cycle, powered 

down the PPMD and 

restarted, problem was 

fixed

11/4/2008 Sensors 

1

Multiple crystals stopped 

oscillating, when one was 

enabled another would 

disable

Grease loading 

slightly too high?

Re cleaned and greased 

crystals

11/4/2008 Horiba 1

Unable to maintain 

setpoint for total flow 

near end of the cycle

High filter loading

Adjusted the valve on the 

total flow pump to allow 

more flow
224

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

11/5/2008 Horiba 1 Sample flow accuracy is 

out of spec

Caused by 

inaccurate dilution 

flow measurement 

(cause of this 

uknown)

Recalibrated VFM

11/6/2008 Horiba 1 Dilution flow is still 

inaccurate Unknown Recalibrated VFM

11/7/2008 Sensors 1

MPS Dilution Flow 

Major audit could not 

pass

Overheated None, cycle voided

11/7/2008 Horiba 1 External compressor 

stopped working Overheated None, cycle voided

11/7/2008 Horiba 1 External compressor 

stopped working Overheated

None, cycle voided, 

installed 2nd compressor 

in parallel

11/12/2008 Sensors 1

PPMD results not 

included in the DS results 

file

Descrepancies in 

the time stamps Never resolved

11/12/2008 Horiba 1 Dilution compressor still 

stoppi ng sometimes Overheating

Connected two 

compressors in parallel, 

so the 2nd will run if the 

first stops

11/12/2008 Sensors 1

Slope on flow for daily 

audits sometimes fails to 

0.97 or 1.03

Monthly 

tolerances too 

tight for daily 

checks

Tolerances relaxed slightly 

for daily checks to save 

time

11/12/2008 Horiba 1

Sample flow check is 

sometimes not within 

tolerance

Monthly 

tolerances too 

tight for daily 

checks

Tolerances relaxed slightly 

for daily checks to save 

time

11/24/2008 Horiba 1 Sample flow accuracy is 

out of spec

Inaccurate 

dilution flow 

measurement

Recalibrated VFM

11/25/2008 AVL 1 MSS failed the external 

DR audit repeatedly

Unknown - all 

internal checks 

passed

Requirement of 

performing external audit 

on daily basis was 

removed
225

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

12/1/2008 Sensors 1 EFM reading inaccurate 

at the end of each test

Solenoid not 

switching 

properly during 

the 1 hour 

autozero, EFM 

zeroing while 

open to flow

Solenoid fixed, auto zero 

disabled?

12/1/2008 Horiba 1

Newest compressor that 

was shipped still stopping 

sometimes

Overheating

Continued to use two 

compressors in parallel, a 

new compressor was 

shipped from Horiba

12/1/2008 Horiba 1

Post processor file size 

too big for Excel (>200 

MB)

Steady state data 

processed at 1Hz 

instead of 10Hz

Continued to use two 

compressors in parallel

12/11/2008 Horiba 1 DCS signal flat lined 

during testing Uknown

Problem did not occur 

again when system was 

restarted

12/15/2008 Horiba 1 TRPM data filed was not 

saved by Java software Unknown None, cycle was void

12/16/2008 Horiba 1 External compressor 

stopped working Overheated None, cycle voided

12/18/2008 Horiba 1 External compressor 

stopped working Overheated None, cycle voided

1/14/2009 Horiba 1 Sample flow accuracy is 

out of spec

Caused by 

inaccurate dilution 

flow measurement 

(cause of this 

uknown)

Recalibrated VFM

1/14/2009 Horiba 1 Dilution flow inaccurate Unknown Recalibrated VFM

1/26/2009 Sensors 2

Semtech DS lost 

communication with 

laptop

Unknown

Couldn't monitor data, but 

data was still recorded on 

compact flash card

1/28/2009 Horiba 2

TPA switched from 

measure to standby at the 

start of the test

Software glitch? Did not occur again
226

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

1/28/2009 Horiba 2

TRPM would switch out 

of filter sample mode 

once 30 seconds had 

elapsed

Faulty software 

logic

Software modified so that 

filter will never switch out 

of sample mode while in 

an NTE event

1/28/2009 Horiba 2

Filter sampling was 

beginning 10 sec after 

start of NTE instead of 5

OBS-2200 is 

delayed 5 

seconds in its 

response

Horiba chose not to make 

any changes

1/30/2009 AVL 2

Error: No dilution air 

available

On-board pump 

was leaking

Pump was replaced with 

new part shipped from 

AVL

2/2/2009 Sensors 2

Semtech DS lost 

communication with 

laptop, could not 

reconnect

Unknown

Powered down unit, 

recovered data the next 

day

2/5/2009 Sensors 2

Semtech DS was losing 

communication with 

laptop

Unknown

Couldn't monitor data, but 

data was still recorded on 

compact flash card

2/18/2009 Horiba 2 External compressor 

stopped working Overheated None, cycle voided

3/2/2009 Horiba 3

Software zero pressure 

transducer function would 

not work after repeated 

attempts

Unknown

Shut down equipment 

attempted it again several 

hours later and it worked

3/3/2009 Horiba 3 External compressor 

stopped working Overheated None, cycle voided

3/5/2009 Sensors 3

Semtech DS lost 

communication with 

laptop

Unknown

Couldn't monitor data, but 

data was still recorded on 

compact flash card

3/6/2009 Horiba 3 External compressor 

stopped working Overheated None, cycle voided

3/23/2009

Horiba 

2200

OBS-2200 software 

locked up during test Unknown

Software worked when 

rebooted, but data for the 

cycle was lost

3/24/2009 Sensors 

DS

Couldn't connect to the 

Semtech DS repeatedly

LAN circuit 

board was likely 

damaged

Unit sent back to Sensors 

for repair
227

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

3/25/2009 Sensors 3

PPMD wouldn't sample 

when NTE trigger 

activates on the Semtech 

DS

Unknown Turning the unit off and 

back on fixed the problem

4/3/2009 AVL 1

There is a voltage offset 

between what the AVL 

unit outputs and the 

Semtech DS reads

Unknown Switched constants

4/13/2009 AVL 2

Power inverter shut down 

during radiated immunity 

test

EMI/RFI Test Replaced bypass pump

4/15/2009 Sensors 2

Communication with the 

laptop dropped out 

repeatedly during bulk 

current injection

EMI/RFI Test Head sent back for repair

4/20/2009 Sensors 2

Communication with the 

laptop dropped out 

during radiated immunity

EMI/RFI Test Replaced with bracket 

from other unit

4/21/2009 AVL 2

Power inverter shut down 

during conducted 

transient test

EMI/RFI Test No corrective action 

taken

4/21/2009 AVL 2

Power inverter shut 

down, then start smoking 

during conducted 

transient

EMI/RFI Test

Unit was shipped back to 

AVL, replaced with other 

unit, problem was not 

observed again

4/21/2009 Sensors 2

PPMD shut down 

completely several times 

during conducted 

transient tests

EMI/RFI Test No corrective action 

taken

4/24/2009 Horiba 3 Flow was erratic during 

bulk current injection test EMI/RFI Test No corrective action taken

4/27/2009 Horiba 3

Exhaust flow was reading 

very high during bulk 

current injection test

EMI/RFI Test No corrective action 

taken

4/28/2009 Horiba 3

Lost communication with 

laptop during radiated 

immunity test

EMI/RFI Test

Inverter replaced with 

backup, test was not 

repeated
228

SwRI Project No. 14936.12

Date PEMS Description Reason Solution

4/28/2009 Horiba 3

Exhaust flow was reading 

very high during radiated 

immunity test

EMI/RFI Test No corrective action 

taken

5/15/2009 AVL 3

AVL PEMS blew a fuse 

when trying to pow er up 

after 1st pressure test

Uknown Replaced with AVL 2

7/15/2009 Sensors 3

PPMD reported a 

barometric pressure 

increase when altitude 

chamber was at vacuum

Constants in the 

software were 

backwards for 

the barometric 

pressure

No corrective action 

taken

7/15/2009 Sensors 3

PPMD could not 

maintain a bypass flow of 

4 slpm

Bypass pump 

was dying

No corrective action 

taken

7/21/2009 AVL 2

Soot concentration 

reading erratic during 

vibration (all orientations)

Vibration Rebooted hardware to 

reconnect

7/22/2009 Horiba 3

L-bracket holding filter in 

ME box broke during 

vibration testing

Vibration

Did not continue with 

more extreme tests to 

prevent damage

7/22/2009 Horiba 3

Total Pi pressure 

transducer would not 

read correctly after 

problem with L-bracket 

occurred

Vibration Test Pressure transducer was 

replaced

7/24/2009 Sensors 2

PPMD could not 

maintain total flow due to 

moisture traps opening

Vibration Test No corrective action 

taken
229

SwRI Project No. 14936.12

APPENDIX B: CRYSTAL BALL OUTPUT FILE DESCRIPTIONS
230

SwRI Project No. 14936.12

EXTRACT DATA FILES

1.0 Simulation Variables

The simulation variables listed in Table 44 were extracted at the completion of the Monte 

Carlo simulation run for each reference NTE event. Crystal Ball classifies variables into two 

categories: assumptions and forecasts. Assumptions are the estimated inputs into the simulation 

model such as the variability indices used to sample each error surface. Assumption variables in 

this study are identified by an “ic” at the beginning of the variable name, or by “Delta” at the 

beginning of the variable name. The “ic

TABLE 44. SIMULATION VARIABLES

” variables are the simulation error model inputs such as 

“01_ic_SS_PM”. The “Delta” variables serve as switches that turn a given error surface on or 

off in the model, e.g. “Delta PM SS”. The Delta switch variables when turned on and off during 

a simulation are applied in post-simulation analysis to determine sensitivity of results to the 

particular error surfaces. Forecasts are values calculated by a forecast formula in the spreadsheet 

cells. Examples of forecast variables used in this study are “001AVL_DePM (g/hp-hr), Method 

1” and “005AVL_Valid DePM (g/hp-hr), Method 2”.

Variable Name Description

001AVL_DePM (g/hp-hr), Method 1 MC Delta PM Method 1 for AVL 

PEMS

001Horiba_DePM (g/hp-hr), Method 1 MC Delta PM Method 1 for Horiba 

PEMS

001Sensors_DePM (g/hp-hr), Method 1 MC Delta PM Method 1 for 

Sensors PEMS

002AVL_DePM (g/hp-hr), Method 2 MC Delta PM Method 2 for AVL 

PEMS

002Horiba_DePM (g/hp-hr), Method 2 MC Delta PM Method 2 for Horiba 

PEMS

002Sensors_DePM (g/hp-hr), Method 2 MC Delta PM Method 2 for 

Sensors PEMS
231

SwRI Project No. 14936.12

003AVL_DePM (g/hp-hr), Method 3 MC Delta PM Method 3 for AVL 

PEMS

004AVL_Valid DePM (g/hp-hr), Method 1 Validation MC Delta PM Method 1 

for AVL PEMS

004Horiba_Valid DePM (g/hp-hr), Method 1 Validation MC Delta PM Method 1 

for Horiba PEMS

004Sensors_Valid DePM (g/hp-hr), Method 1 Validation MC Delta PM Method 1 

for Sensors PEMS

005AVL_Valid DePM (g/hp-hr), Method 2 Validation MC Delta PM Method 2 

for AVL PEMS

005Horiba_118_Valid DePM (g/hp-hr), Method 2 Validation MC Delta PM Method 2 

for Horiba PEMS

005Sensors_Valid DePM (g/hp-hr), Method 2 Validation MC Delta PM Method 2 

for Sensors PEMS

006AVL_Valid DePM (g/hp-hr), Method 3 Validation MC Delta PM Method 3 

for AVL PEMS

01_ic_SS_PM

Random Sampling Variability 

Index for SS PM Error Surface, 

applied to AVL, Horiba and 

Sensors PEMS

02_ic_TR_PM

Random Sampling Variability 

Index for Transient PM Error 

Surface, applied to AVL, Horiba 

and Sensors PEMS

04_ic_Atm.Pres_PM_AVL

Random Sampling Variability 

Index for PM Atmospheric Pressure 

Error Surface, applied to AVL 

PEMS

05_ic_Amb.Temp_PM_AVL

Random Sampling Variability 

Index for PM Temperature Error 

Surface, applied to AVL PEMS

07_ic_SS_CO Random Sampling Variability 
232

SwRI Project No. 14936.12

Index for SS CO

10_ic_Pressure_CO Random Sampling Variability 

Index for CO Pressure

11_ic_Temperature_CO Random Sampling Variability 

Index for CO Temperature

13_ic_SS_NMHC Random Sampling Variability 

Index for SS NMHC

14_ic_TR_NMHC Random Sampling Variability 

Index for Transient NMHC

16_ic_Pressure_NMHC Random Sampling Variability 

Index for NMHC Pressure

17_ic_Temperature_NMHC Random Sampling Variability 

Index for NMHC Temperature

19_ic_NMHC_Ambient Random Sampling Variability 

Index for Ambient NMHC

20_ic_SS_flow Random Sampling Variability 

Index for SS Exhaust Flow

21_ic_TR_Flowrate Random Sampling Variability 

Index for Transient Exhaust Flow

22_ic_Pulsation_flow Random Sampling Variability 

Index for Exhaust Flow Pulsation

23_ic_Swirl_flow Random Sampling Variability 

Index for Exhaust Flow Swirl

25_ic_Radiation_Exhaust Flow

Random Sampling Variability 

Index for Exhaust Flow EMI/RFI 

Radiation

27_ic_Temperature_Exhaust Flow

Random Sampling Variability 

Index for Exhaust Flow 

Temperature

28_ic_Pressure_Exhaust Flow Random Sampling Variability 
233

SwRI Project No. 14936.12

Index for Exhaust Flow Pressure

29_ic_TR_Torque Random Sampling Variability 

Index for Dynamic Torque

30_ic_Torque_DOE

Random Sampling Variability 

Index for Torque Design of 

Experiments Testing

31_ic_Torque_Warm Random Sampling Variability 

Index for Torque Warm-up

32_ic_Torque_IP

Random Sampling Variability 

Index for Torque Independent 

Parameters Humidity and Fuel

34_ic_Torque_Interpolation Random Sampling Variability 

Index for Torque Interpolation

35_ic_Torque_Engine Manufacturers

Random Sampling Variability 

Index for Torque Engine 

Manufacturers

42_ic_Fuel_Engine Manufacturers

Random Sampling Variability 

Index for Fuel Engine 

Manufacturers

43_ic_TR_Speed Random Sampling Variability 

Index for Dynamic Speed

44_ic_TR_Fuel Rate Random Sampling Variability 

Index for Dynamic Fuel Rate

45_ic_SS_CO2 Random Sampling Variability 

Index for SS CO2

46_ic_TR_CO2 Random Sampling Variability 

Index for Transient CO2

49_ic_Temperature_CO2 Random Sampling Variability 

Index for CO2 Temperature

Delta PM SS Model switch controlling PM SS 

error surface application
234

SwRI Project No. 14936.12

Delta PM Transient Model switch controlling PM 

Transient error surface application

Delta PM Atmospheric Pressure

Model switch controlling PM 

Atmospheric Pressure error surface 

application

Delta PM Ambient Temperature

Model switch controlling PM 

Ambient Temperature error surface 

application

Delta CO SS Model switch controlling CO SS

error surface application

Delta CO Atmospheric Pressure

Model switch controlling CO 

Atmospheric Pressure error surface 

application

Delta CO Ambient Temperature

Model switch controlling CO 

Ambient Temperature error surface 

application

Delta NMHC SS Model switch controlling NMHC 

SS error surface application

Delta NMHC Transient Model switch controlling NMHC 

Transient error surface application

Delta NMHC Atmospheric Pressure

Model switch controlling NMHC 

Atmospheric Pressure error surface 

application

Delta NMHC Ambient Temperature

Model switch controlling NMHC 

Ambient Temperature error surface 

application

Delta Ambient NMHC Model switch controlling Ambient 

NMHC error surface application

Delta Exhaust Flow SS Model switch controlling Exhaust 

Flow SS error surface application

Delta Exhaust Flow Transient Model switch controlling Exhaust 

Flow Transient error surface 
235

SwRI Project No. 14936.12

application

Delta Exhaust Flow Pulsation

Model switch controlling Exhaust 

Flow Pulsation error surface 

application

Delta Exhaust Flow Swirl

Model switch controlling Exhaust 

Flow Swirl error surface 

application

Delta Exhaust EMI/RFI Model switch controlling Exhaust 

EMI/RFI error surface application

Delta Exhaust Temperature

Model switch controlling Exhaust 

Temperature error surface 

application

Delta Exhaust Pressure Model switch controlling Exhaust 

Pressure error surface application

Delta Dynamic Torque Model switch controlling Dynamic 

Torque error surface application

Delta Torque DOE Testing

Model switch controlling Torque 

DOE Testing error surface 

application

Delta Torque Warm-up Model switch controlling Torque 

Warm-up error surface application

Delta Torque Humidity Model switch controlling Torque 

Humidity error surface application

Delta Torque Interpolation

Model switch controlling Torque 

Interpolation error surface 

application

Delta Torque Engine Manuf

Model switch controlling Torque 

(Engine Manufacturer) error 

surface application

Delta Fuel Engine Manuf

Model switch controlling Fuel 

(Engine Manufacturer) error 

surface application
236

SwRI Project No. 14936.12

Delta Dynamic Speed Model switch controlling Dynamic 

Speed error surface application

Delta Dynamic Fuel Rate Model switch controlling Dynamic 

Fuel Rate error surface application

Delta CO2 SS Model switch controlling CO2 SS

error surface application

Delta CO2 Transient Model switch controlling CO2 

Transient error surface application

Delta CO2 Ambient Temperature

Model switch controlling CO2 

Ambient Temperature error surface 

application

2.0 Statistics

Descriptive statistics summarizing the values obtained during a single reference NTE event 

simulation are provided in Table 45.

TABLE 45. DESCRIPTIVE STATISTICS FOR SIMULATION VARIABLES

Statistic Definition

Trials Number of times the simulation was 

repeated

Mean Arithmetic average

Median The value midway between the smallest 

value and the largest value

Mode Value that occurs most often

Standard Deviation Measurement of variability of a 

distribution. The square root of the 

variance
237

SwRI Project No. 14936.12

Variance The average of the squares of the 

deviations of a number of values from their 

mean

Skewness A measure of the degree of deviation of a 

distribution from the norm of a symmetric 

distribution

Kurtosis A measure of the degree of peakedness of a 

distribution

Coefficient of Variability Standard deviation/Mean

Minimum Smallest value

Maximum Largest value

Range Width Largest value – smallest value

Mean Standard Error Standard deviation of the distribution of 

possible sample means

3.0 Percentiles

Percentiles are the probability of achieving values below a particular percentage in the 

following increments: 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 

and 100%. Percentiles are computed for each of the simulation variables described in Table 44.

4.0 Sensitivity Data

Sensitivity data are provided by computing the rank correlation coefficient for all error 

surfaces and all simulation variables. The EXTRACT data file contains the absolute value of 

the rank correlation. In post-simulation processing, values of control variable Delta PM SS in 

the simulation results were applied to dichotomize the data.

5.0 Trial Values

The value for all simulation variables is provided at each trial of the simulation.
238

SwRI Project No. 14936.12

REPORT FILES

1.0 Report Summary

This section includes the simulation start date and time, stop date and time, number of 

trials run, sampling type (Monte Carlo), random seed used, and run statistics.

2.0 Forecasts

Descriptive statistics, percentiles, and a frequency histogram are provided for forecast 

variables 001AVL_DePM (g/hp-hr), Method 1 through 006AVL_Valid DePM (g/hp-hr), Method 

3 (see Table 44).

3.0 Assumptions

Descriptive statistics, percentiles, distribution parameters, and a distribution chart are 

provided for assumption variables 01_ic_SS_PM through 49_ic_Temperature_CO2 (see Table 

44).

4.0 Sensitivity Charts

Sensitivity charts are provided for forecast variables 001AVL_DePM (g/hp-hr), Method 1

through 006AVL_Valid DePM (g/hp-hr), Method 3 (see Table 44). Crystal Ball calculates 

sensitivity by computing rank correlation coefficients between every assumption (error surface) 

and forecast (delta BSPM emissions) while the simulation is running. Positive rank correlations 

indicate that an increase in the assumption is associated with an increase in the forecast. The 

larger the absolute value of the rank correlation the stronger the relationship.
239

SwRI Project No. 14936.12

The sensitivity charts developed during the MC simulation are displayed as ‘Contribution 

to Variance” charts which are calculated by squaring the rank correlation coefficients for all 

assumptions used in a particular forecast and then normalizing them to 100%. Figure 180

displays a sensitivity chart for the AVL delta PM Method #1. The assumptions with the highest 

contribution to variance (in absolute value) are plotted at the top of the chart. This example 

shows a typical dominant effect of the PM SS error surface represented by the 79.5% negative 

effect of Delta PM SS. As seen in the example in Figure 180, as you increase the SS PM there is 

an increase in the delta PM Method #1 values represented by the 7.6% positive effect of 

ic_SS_PM, and as you increase the torque warm-up there is a decrease in the delta PM Method 

#1 values represented by the 2.8% negative effect of ic_Torque_Warm. Only the top eight 

assumptions are plotted in this sensitivity chart.
240

SwRI Project No. 14936.12

FIGURE 180. SENSITIVITY CHART FOR AVL DELTA PM METHOD 1
241

SwRI Project No. 14936.12

APPENDIX C: MONTE CARLO SPREADSHEET COMPUTATIONS
242

SwRI Project 14936.12

1.0 DESCRIPTION OF ASSUMPTIONS

The following assumptions were made in running the Monte Carlo model:

• Only one reference NTE event can be run at a time through the Monte Carlo

simulation workbook. However, NTE event cases can be stacked and run in a batch 

mode.

• Uniform (1 second in duration) time steps are used in the reference NTE events.

• Standard format and engineering units for reference NTE data established for the 

project are observed, and applied to the reference NTE event before the NTE event is 

entered in the Error Model workbook for Monte Carlo simulation.

• Any wet – dry matter conversions, if not negligible, have been performed on the 

appropriate reference NTE event values before the reference NTE event was entered 

in the Error Model workbook for Monte Carlo simulation. No wet – dry conversions 

are performed in the workbook.

• Any reference NTE event normalizations to produce similar emissions brake-specific 

results from the three emissions calculation methods have been appropriately 

performed before the reference NTE event was entered in the Error Model workbook 

for Monte Carlo simulation. No normalizations among the three methods are 

performed in the workbook.

• PM emissions models for three calculation methods are computed for the AVL 

PEMS. Only calculation methods 1 and 2 are computed for the Horiba and Sensors 

PEMS.

• Error surface models and supporting data were approved by the Steering Committee.

• The error model spreadsheet has been correctly implemented, and its interaction with 

Monte Carlo tools like Crystal Ball is correctly understood.

• Random number generation by a Monte Carlo tool like Crystal Ball is correct.
243

SwRI Project 14936.12

• Convergence of the completed MC simulation was processed and checked outside of 

the simulation workbook. Benchmark checks on the convergence calculations were 

made using a SAS

2.0 WORKSHEET DESCRIPTIONS

2.1 Macro Description

The Macro can be viewed in the Excel spreadsheet ‘Batch Control’ with the menu 

selections Tools>Macros>Macro1>Edit. The purpose of Macro1 is to control NTE event batch 

processing of stacked cases. For each NTE event case processed, the macro expedites clearing

extra cells below the reference NTE event in the spreadsheet ‘Error Model’ Methods worksheet

and deletes extra rows in the Delta error worksheets. The macro also performs Mode 0 

calculations and stores resultant 'ideal emissions' values for application in subsequent Monte 

Carlo simulation.

The user must begin with the starter version of the ‘Error Model’ Excel file which has 

300 rows of equations in columns X - CF and in rows 52 - 351 in the Methods worksheet. The 

starter spreadsheet also has 300 rows of equations below charts in columns B – F, or B – L, in 

applicable Delta worksheets. The user (when not under automatic batch control) copies the

reference NTE event into columns A – V, row 52 and down, in the Methods worksheet. It is can 

then be confirmed that cell J45 in the Methods worksheet displays the correct number of rows of 

the reference NTE event.

Macro execution can be accomplished through the menu selections 

Tools>Macros>Macro1>Run. Note that this macro clears cells without deleting rows in the 

Methods worksheet, and deletes rows in the Delta worksheets. This macro will not work if the 

reference NTE event has only one row. For a reference NTE event with exactly two rows, this 

macro will corrupt the second "check" values in columns B-F type Delta worksheets. Check 

values are not used in the simulation, but are provided as a diagnostic aid. Apply the macro for 

reference NTE events with no more than 300 rows.

computer program.

The reader can follow the description of execution that follows by viewing the macro and 

observing the comment rows provided throughout the macro text. In execution, the macro first 

reads the contents of J45 in the Methods worksheet. It uses the number of rows in the reference 
244

SwRI Project 14936.12

NTE event defined by J45 to determine how many rows to clear and delete in the spreadsheet. It 

checks that the number of rows is between 2 and 299, inclusive. It will also execute correctly for 

300 rows.

Next, the macro clears cell contents in columns X – CF below the reference NTE event in 

the Methods worksheet. Note the macro, as written, will not execute properly if the starter 

spreadsheet has been revised with row insertion or deletion in certain areas of the spreadsheet. 

As written, the macro initiates in cell X52, counts down through the NTE Event rows, and clears 

contents in the range from there in column X through cell CF351.

Next, the macro deletes extra rows below the reference NTE event, where applicable, for 

example in Delta worksheet 07. For Delta worksheet 07 it initiates in cell B79 and counts down 

through the rows of the reference NTE event to the first row to be deleted. It selects the range of 

rows from there down through row 378, deletes the rows, copies some equations and a value to 

the last row in the range the charts use, and returns the cursor to cell F68 leaving the display 

more or less centered on the charts in the worksheet.

Subsequently, the macro performs similar operations in other Delta worksheets; however, 

the initiating cell and final row differ among the worksheets. The Delta worksheets processed in 

this way are 7, 10, 11, 16, 17, 20, 21, 22, 23, 29, 30, 43, 44, 45, 46 and 49.

Following the row deletion operations in the Delta worksheets, or directly when the 

reference NTE event has 300 rows, the macro prepares for the Mode 0 (ideal emissions) 

calculation. First, in the Methods worksheet it copies the equations in row 52, columns X 

through CF, to the last row in the reference NTE event. This clears any errors introduced in the 

last row; however, it assumes that row 52 is correct. The last cell in column AC (∆t) is cleared 

for aesthetics, since the ∆t values are not applied in the model calculations.

The Mode 0 calculation is performed by the macro by changing the value in cell A6 of 

the Summary worksheet to 0. Then in the Methods worksheet, the values from cells CU22 

through CU30 are pasted (values only) to cells O22 through O30 where they are referenced by 

formulas during Monte Carlo simulation. The macro changes the value of A6 in the Summary 

worksheet to 2 in preparation for the Monte Carlo simulation, and moves the cursor to cell CT18 

of the Methods worksheet.
245

SwRI Project 14936.12

Additional comments regarding the macro operation are presented in the following 

section descriptions of the model spreadsheet.

2.2 Worksheet 1: ErrorControl

The ErrorControl worksheet of the Error Model workbook implements 31 logic switch 

functions. The user enters a numerical “1” in column AD in each row corresponding to error 

surfaces to be included in the calculation. A numerical “0” is applied to error surfaces to be 

excluded in the calculation. Corresponding random variables for error-surface on-off switch 

random effects sensitivity modeling are implemented in column W under Crystal Ball control.

Error surfaces are numbered 1 through 49. The numbered error surfaces are defined in 

columns A – C, and information pertinent to their usage is presented in columns E – V of the 

worksheet. Column E displays warning messages when an unusual value is monitored in column 

D.

The control switch elements in the worksheet are deliberately placed on rows in the 

worksheet corresponding to the error surfaces to expedite equation checking in the Methods 

worksheet where the control switch variables are applied in conjunction with error surfaces from 

the correspondingly numbered “Delta” worksheets

The numbered error surfaces and time alignment controls that have been implemented are 

defined in the following Table 46. 
246

SwRI Project 14936.12

TABLE 46. ERROR SURFACES USED IN SIMULATION

Component No. Error Surface

Delta PM 1 Delta PM SS

2 Delta PM Transient

4 Delta PM Atmospheric Pressure

5 Delta PM Ambient Temperature

Delta CO 7 Delta CO SS

10 Delta CO Atmospheric Pressure

11 Delta CO Ambient Temperature

Delta NMHC 

NMHC = 0.98*THC

13 Delta NMHC SS

14 Delta NMHC Transient

16 Delta NMHC Atmospheric Pressure

17 Delta NMHC Ambient Temperature

19 Delta Ambient NMHC

Delta Exhaust Flow 20 Delta Exhaust Flow SS

21 Delta Exhaust Flow Transient

22 Delta Exhaust Flow Pulsation

23 Delta Exhaust Flow Swirl

25 Delta Exhaust EMI/RFI

27 Delta Exhaust Temperature

28 Delta Exhaust Pressure

Delta Torque 29 Delta Dynamic Torque

30 Delta Torque DOE Testing

(Interacting Parameters Test)

31 Delta Torque Warm-up
247

SwRI Project 14936.12

(Interacting Parameters Test)

32 Delta Torque Humidity / Fuel

(Independent Parameters Test)

34 Delta Torque Interpolation

35 Delta Torque Engine Manufacturers

Delta Fuel 42 Delta Fuel Engine Manufacturers

Delta Speed 43 Delta Dynamic Speed

Delta Fuel Rate 44 Delta Dynamic Fuel Rate

Delta CO2 45 Delta CO2 SS

46 Delta CO2 Transient

49 Delta CO2 Ambient Temperature

The thirty-one (31) error surfaces that have been implemented are included or excluded 

by the controls numbered 1 – 49 identified in Table 46. When all 31 error controls are on 

(included in calculation), the sum of column D in the worksheet ErrorControl is 31.

2.3 Worksheet 2: Summary

The Summary worksheet in the Error Model workbook comprises input mode control in 

rows 4 – 10 and output summary in rows 88 and 119. Other rows in this worksheet are available 

for diagnostic purposes.

The calculation mode control is accomplished with cell A6 where the user normally 

confirms that a numerical value of “2” is designated. Mode 2 designates emissions calculation 

with all errors applied. Mode 1 corresponds to a calculation of emissions with all errors applied 

except environmental errors. Mode 0 designates an “ideal” emissions calculation with no errors

applied. In Monte Carlo error model simulation performed in this study Mode 2 was used.

Mode 0 is used prior to Monte Carlo simulation to generate the “ideal” emissions for a 

given reference NTE event. The Mode 0 values are calculated by entering a value of “0” in cell 

A6. The Mode 0 calculation and subsequent storing of the “ideal” emissions results may be 

accomplished manually (as described above) or by exercising a provided macro. The macro 
248

SwRI Project 14936.12

automatically sets the value in cell A6 to zero, calculates and saves the “ideal” emissions values, 

and returns the value in A6 to “2” in preparation for the Monte Carlo simulation. The locations 

where the reference NTE event must be entered manually, and the locations where the “ideal” 

emissions must be saved (done automatically if the macro is used) are described in the Methods 

worksheet section.

Mode 1 in cell A6 is not typically used but can be applied for diagnostic purposes.

The output summary section of the Summary worksheet in rows 88 and 119 presents 

numerically and descriptively labeled outputs of the emissions and emissions error calculations.

In the output summary, the cells that are highlighted in turquoise color are designated by 

Crystal Ball as “Forecast” (or output) random variables. 

A total of 14 outputs (“Forecasts”) are designated in the Summary worksheet rows 88 and 

119 covering the number of output values from PM emission, three calculation methods 

(Methods 1, 2 and 3) for the AVL PEMS and two methods (Methods 1 and 2) for the Horiba and 

Sensors PEMS, for the full error model and for the validation model (designated Valid in 

Summary worksheet variable labels). All of these “Forecasts” are provided in units of grams/hphr. This variety of calculations was accomplished with the Methods worksheet.

2.4 Worksheet 3: Methods

The Methods worksheet of the Error Model workbook comprises the following areas:

• Notes and diagnostic guides are located principally in rows 1 – 21 in columns A – CF, 

continuing on row 5 through column DD.

• Reference NTE event data are located in rows 35 - 351 of columns A – W. Actual 

reference NTE event data must be entered manually (or automatically under batch 

control) starting on row 52 in columns A – V. Cell W52 data must be entered, and is 

provided for special case study where the Method 3 flow-weighted PM concentration 

may differ from Methods 1 and 2. One to 300 rows of reference NTE event data are 

allowed. Uniform (one second interval) time steps are assumed represented by the 

reference NTE data.

• Parameters calculated are located in rows 35 – 351 of columns X – CF. The number of 

rows of these parameter equations must match the number of rows in the reference NTE 
249

SwRI Project 14936.12

event. Excess cells in these columns may be cleared manually or automatically during 

execution of the macro.

• Mode 0, Ideal Emissions for this reference NTE event are stored in column O rows 22 –

30 (either manually or automatically by the macro). Related data on the same rows are 

located in columns CT – DD.

• Input ic random variable distributions (Crystal Ball uses the terminology “Assumptions” 

for these inputs) are located in rows 26 – 32 of columns AG - CC.

• Emissions calculations by three methods are located in rows 6 - 81 of columns CH – CJ 

(Method 1), CL – CN (Method 2) and CP (Method 3). This part of the worksheet 

calculates full model and validation model.

2.5 Methods Worksheet:Notes and Diagnostic Guide

In rows 1 – 22 for columns A – CF, several descriptive labels and references are defined 

for use in navigating through the worksheet. Row 5, columns A-DD, contains column 

identification numbers referenced in rows 7 through 22 (depending on the column). For 

example, in column H the values 65 on row 8 indicates that the values in column H (rows 52 and 

following rows) are applied in column 65 (BM) labeled on row 5. If the user scrolls to cell 

BM52 it is observed that the spreadsheet formula in the cell refers to values from column H. The 

information in the notes and diagnostic guide was not applied by the spreadsheet in any of the 

emissions calculations. It was included with the intent to simplify diagnostics by providing 

information on locations where spreadsheet values were applied elsewhere in the spreadsheet. 

Outside the areas indicated above, some other notes, comments and diagnostic guides may be 

found in other areas of the spreadsheet.

2.5.1 Methods Worksheet: Reference NTE Event

The reference NTE event used in the simulation was entered in rows 35-351 of columns 

A – W. Actual reference NTE event data must be entered manually starting on row 52 in 

columns A – V. Cell W52 data must be entered, and is provided for special case study where the 

Method 3 flow-weighted PM concentration may differ from Methods 1 and 2. A minimum of 

one and a maximum of 300 rows of reference NTE event data are allowed. Equal time steps (1 
250

SwRI Project 14936.12

second intervals) are assumed in the reference NTE data rows. The standard format and 

engineering units of reference NTE event data established for this project must be observed. 

These are described in the column headings on rows 47 – 51, columns A – V.

2.5.2 Methods Worksheet: Parameters

Parameters applied in the three emissions methods are calculated in rows 35 – 351 of 

columns X – CF. The number of rows of these parameter equations must match the number of 

rows in the reference NTE event. Excess cells in these columns may be cleared manually or 

automatically during execution of the macro. 

The formulas applied in rows 52 and down in columns X – CF have been produced by 

normal edit-copy (typically of row 52 in these columns) and edit-paste to rows 53 and following 

rows in these columns. The ∆t values displayed in column AC are not used in any calculation, 

but are displayed so a user can confirm uniform reference NTE event time sampling. The last 

cell in column AC can be cleared (done automatically by the macro). Note that excess cells in 

these columns must be cleared, and row deletion operations should not be applied since this 

would affect other areas in the Methods worksheet. 

Certain sums are performed in several columns over the Parameter rows (range of the 

reference NTE event). These are accomplished in row 46 in columns AI, AW, AX, BO, BQ, BU

and CA. Flow-weighted PM concentration SS and TR errors are consolidated, with and without 

environmental errors, in the area of cells AN40:AB47. Certain constants applied in the 

calculation are stored in cells AW42, BC42, BI42, BP40 and BP42. Other constants or 

conversion factors are incorporated numerically in spreadsheet formulas. Typical of these is 

“0.01” to convert a percentage to a fraction.

Specific parameters or variables are calculated in the various columns for application in 

all three methods, full model and validation model. Table 47 lists the parameters used in the 

Methods worksheet, the columns where they are computed and a brief description of the 

parameters.

TABLE 47. METHODS WORKSHEET PARAMETER COLUMN DESCRIPTIONS

Methods Worksheet Parameters Column Descriptions

Subject Column Description

Engine operating state 

percentages

X – AB Convert NTE Event variables to percentages: speed, torque, fuel rate, 

exhaust flow

∆Time AC Displays ∆t between NTE Event rows
251

SwRI Project 14936.12

NMHC AD Calculate NMHC ppm as 0.98 of THC ppm

Fuel Rate AE Calculate fuel rate g/s based on fuel density of 851 g/L

Exhaust Flow 

Calculations

AF Convert exhaust flow SCFM to mol/s

AG Sum exhaust flow errors from Delta tabs 20, 21, 22, 23, 25 27 and 28 

expressed in % of mol/s maximum. Respective ErrorControl tab 

switches are applied.

AH Convert the total exhaust flow error in % of maximum mol/s to mol/s

AI Add the mol/s exhaust flow error to the exhaust flow in mol/s. Mode 

control logic is applied.

Speed with error AJ Add engine speed error from Delta tab 43 expressed as % of engine 

range converted to engine speed in rpm. Mode control logic and 

ErrorControl switch are applied.

Fuel rate with error AK Combine Delta tab 42 Fuel (engine manufacturer) with fuel rate from 

Delta tab 44 expressed as % of maximum fuel rate converted to g/s to 

engine fuel rate in g/s. Mode control logic and ErrorControl switch are 

applied.

Torque AL Sum torque errors from Delta tabs 29, 30, 31, 32, 34 expressed as % of 

peak torque, and from Delta tab 35 expressed as % of NTE point 

torque converted to % of peak torque. ErrorControl switches are 

applied.

AM Add the total torque error expressed as % of peak torque converted to 

N·m to engine torque in N·m. Mode control logic is applied.

PM, μg/mol AN42, AN43 Sum PM SS and TR errors from Delta tabs 1 and 2 for AVL PEMS 

expressed as μg/mol, AN 42 for Methods 1 and 2, AN43 for Method 3. 

ErrorControl switches are applied.

AO42 Sum PM SS and TR errors from Delta tabs 1 and 2 for Horiba PEMS 

expressed as μg/mol, for Methods 1 and 2. ErrorControl switches are 

applied.

AP42 Sum PM SS and TR errors from Delta tabs 1 and 2 for Sensors PEMS 

expressed as μg/mol, for Methods 1 and 2. ErrorControl switches are 

applied.

AQ42, AQ43 Sum AVL flow-weighted PM concentration to all errors except 

environmental PM errors, AQ 42 for Methods 1 and 2, AQ43 for 

Method 3. Mode control logic is applied. 

AR42 Sum Horiba flow-weighted PM concentration to all errors except 

environmental PM errors. Mode control logic is applied.

AS42 Sum Sensors flow-weighted PM concentration to all errors except 

environmental PM errors. Mode control logic is applied.
252

SwRI Project 14936.12

AQ46, AQ47 Sum AVL flow-weighted PM concentration to all errors including 

environmental PM errors Delta tabs 4 and 5, AQ 46 for Methods 1 and 

2, AQ47 for Method 3. Mode control logic is applied.

AR46 Same formula as AR42 as no environmental PM error model applied 

to Horiba flow-weighted PM concentration.

AS46 Same formula as AS42 as no environmental PM error model applied to 

Sensors flow-weighted PM concentration.

Speed · Torque AW Form product of Speed (rpm, all errors case, column AJ) and Torque 

(N·m, all errors case, column AM) for application in Methods 1 and 3. 

Convert rpm to radians/sec with 2πradians/revolution, minutes to 

seconds with 60sec/min, N·m/sec to watt hr with 3600Joules/watt hr, 

and watt to kw with 1000w/kw.

AX Form product of Speed (rpm, no errors for validation case, column O) 

and Torque (N·m, no errors for validation case, column T) for 

application in Methods 1 and 3. Convert rpm to radians/sec with 

2πradians/revolution, minutes to seconds with 60sec/min, N·m/sec to 

watt hr with 3600Joules/watt hr, and watt to kw with 1000w/kw.

CO and ∆CO, % AY Sum environmental CO errors including errors from Delta tabs 10 and 

11.

AZ Sum other CO errors. Error from Delta tab 7 is the only one 

developed.

BA Add the total CO errors expressed as % to engine CO in %. Mode 

control logic is applied.

NMHC and ∆NMHC, 

ppm

BE Sum environmental NMHC errors including errors from Delta tabs 16, 

17 and 19.

BF Sum other NMHC errors including errors from Delta tabs 13 and 14.

BG Add the total NMHC errors expressed as ppm to engine NMHC in 

PPM. Mode control logic is applied.

CO2 and ∆CO2, % BK Sum environmental CO2 errors. Error from Delta tab 49 is the only 

one developed.

BL Sum other CO2 errors including errors from Delta tabs 45 and 46.

BM Add the total CO2 errors expressed as % to engine CO2 in %. Mode 

control logic is applied.

Exhaust Flow ·

[ NMHC + ( CO+CO2 

BO

) 

] / [fuel mass flow rate / 

Speed · Torque ]

Form product of hydrocarbons fraction plus CO and CO2 fractions (all 

errors case, columns BG, BA and BM) and exhaust flow (mol/s, 

column AI) divided by ratio of fuel rate (g/s, all errors case, column 

AK) to speed-torque product (all errors case, column AW) for

application in PM Method 2. 
253

SwRI Project 14936.12

BQ Form product of NMHC fraction plus CO and CO2 fractions (all errors 

case, columns BG, BA and BM) and exhaust flow (mol/s, column AI) 

divided by ratio of fuel rate (g/s, no errors case, column AE) to speedtorque product (no errors case, column AX) for application in PM 

Method 2 validation. 

NMHC + ( CO+CO2 ) BS Form sum of NMHC fraction plus CO and CO2 fractions (all errors 

case, columns BG, BA and BM) for application in Method 3. 

Fuel Rate / [ NMHC + ( 

CO+CO2 

BU

) ]

Form quotient, Fuel Rate (g/s, all errors case, column AK) divided by 

sum of NMHC fraction plus CO and CO2 fractions (all errors case, 

column BS) for application in Method 3. 

Fuel Rate / [ NMHC + ( 

CO+CO2 

CA

) ]

Form quotient, Fuel Rate (g/s, no errors case, column AE) divided by 

sum of NMHC fraction plus CO and CO2 fractions (all errors case, 

column BS) for application in Method 3 validation. 

2.5.3 Methods Worksheet: Mode 0 Ideal Emissions

For the reference NTE event in rows 52 and down in columns A – V, an ideal emissions

value must be calculated and stored for application in the emissions difference calculations. The 

ideal case can be calculated either manually, or automatically by the macro. Following the 

calculation, the ideal values are stored by edit-copy edit-paste-special-values operation to the

cells in column O, rows 22 – 30. The manual operations described below are performed 

automatically by the macro, if executed, after manually entering the reference NTE event.

After manually entering the reference NTE event to be simulated and checking that the 

number of rows of equations in the Parameters section matches the rows in the reference NTE 

event, a numerical “0” can be entered in cell A6 of the Summary worksheet. The Methods 

worksheet should have calculated Mode 0 results using the reference NTE event. If error 

messages like “#VALUE or #DIV/0!” are displayed, there is probably still a mismatch between 

the rows of the reference NTE event and Parameter equations. When calculated properly (with 0 

in Summary A6), the values displayed in the Methods worksheet columns CU, CV and DB will 

be equal on each of the rows 22 – 30. The values in column CX are not yet equal (unless 

previously calculated and stored for this reference NTE event) because they reflect the values 

stored in Methods worksheet column O, rows 22 – 30. The next manual step is to edit-copy 

column CU, rows 22 - 30, and store the values by edit-paste-special-values in column O, rows 22 

- 30. Now in rows 22 - 30 the columns CU, CV, CX and DB should be equal. The final step is 

to return to Summary worksheet cell A6 and change the value from 0 to 2. At this point the 

spreadsheet could be run in Monte Carlo simulation to produce properly sampled values. 

However, if the user desires to monitor charts provided in the Delta worksheets during the 

simulation, further row-matching to the reference NTE event is required in most of the Delta 

worksheets.
254

SwRI Project 14936.12

The manual operations described in the previous paragraph are intended to explain how 

the Mode 0 ideal emissions are calculated and stored for use in the Monte Carlo simulation when 

∆emissions values are calculated using the ideal emissions results stored in O22 – O30. The 

reference NTE event must be entered with an operation such as a manual edit-copy and editpaste or edit-paste-special-values operation. The macro automatically performs the mode 0 

calculation, stores the mode 0 results in O22 – O30, and changes Summary A6 back to mode 2. 

The macro also deletes extraneous rows from all the appropriate Delta worksheets so the 

charts therein display properly. It is important to copy the reference NTE event into a fully 

‘loaded’ starter file with equations filled on 300 rows in the Parameters area, and with full 300 

row complement of equation-rows in each of the appropriate Delta worksheets for the macro to 

modify the spreadsheet properly.

2.5.4 Methods Worksheet: Input ic Random Variable Distributions

Probability distribution parameters are applied, and simulation trial values of the inputs 

are generated in rows 26 – 32 of columns AG - CC. Rows 26 and 27 are used to input 

distribution parameters. Rows 28 and 29 contain descriptive labels brought from the appropriate 

Delta worksheet. Row 30 is an information-only number, row 31 contains the name label 

applied in Monte Carlo simulation to the input ic, and row 32 is where the Monte Carlo 

simulation tool places generated randomly-sampled values during simulation. The values in row 

32 are referenced by formula in the respective Delta worksheets where they are used for 

interpolation on the error surfaces. 

The Monte Carlo tool in Crystal Ball uses the terminology “Assumptions” for these 

inputs. Two distribution forms are applied: truncated normal (Gaussian), and discrete uniform. 

For the normal distribution, the applied standard deviation is in row 27. In Crystal Ball, the 

standard deviation cell on row 27 and the label cell on row 31 were referenced by equation in the 

Crystal Ball assumption setup window, the mean was input as 0, and the distribution was 

truncated at -1.414319083 and at +1.414319083. Since all the truncated normal ic distributions 

are identical (although the sampled trial values from each will be random in the Monte Carlo 

simulation), the Crystal Ball define-copy data and define-paste data operations were applied to 

define the truncated normal distributions for other ic

For the discrete uniform distributions, the minimum discrete value (1 in all cases) was 

applied in row 26, the maximum discrete value was applied in row 27 and the other row 

descriptions are the same as before. Again, one of these inputs was setup with Crystal Ball 

variables once the first one had been 

defined.
255

SwRI Project 14936.12

“define assumption” and then applied with Crystal Ball define-copy data and define-paste data 

operations to other ic cells on row 32 where a discrete uniform distribution was applied. When 

Crystal Ball “Assumptions” were defined, Crystal Ball colored each input cell bright green

During a Monte Carlo simulation, the Monte Carlo tool (e.g. Crystal Ball) placed a 

numerical value in each of the ic cells on row 32. Then the spreadsheet was exercised to perform 

interpolations in all the Delta worksheets. The resulting error sample values for the entire 

reference NTE event were returned to the Methods worksheet Parameters area, and then the 

Methods worksheet Emission Calculations section computes ∆emissions using three methods, 

full model and validation to generate one set of the 14 output values described in the Summary 

section. The simulation tool stores the set of random input values from row 32 as well as the 

output values in an Excel data base from which the corresponding sets of values can later be 

extracted. Once each trial was completed, the simulation tool randomly sampled a second set of 

input values from the respective probability distributions, placed the values in the cells on row 

32, exercised the spreadsheet again, stored the input and output values, and went to a third trial, 

etc. Typically 40,000 to 65,000 trials, depending on the reference NTE event, were used in this 

project with this Error Model workbook.

Note that there are three ways the user can control the effect of the ic values in the 

emissions calculations:

Mode control in Summary A6,

Include / exclude switches in ErrorControl column AD, and

Specification of input random variables (“Assumptions”) and their probability 

distributions in the Methods worksheet row 32.

These three ways of controlling the ic

1. Mode 0 - no errors included

values are independent, but the effects are interdependent 

as follows. Mode control determines what categories of errors are added into the calculations. 

Mode controls categories of errors are classified as:

2. Mode 1 - “all” but ‘environmental’ errors included

3. Mode 2 - “all” errors added into the calculations

“All” in this context represents those error surfaces turned on by the switches in the ErrorControl 

worksheet. The input random variable distribution controls the distribution of the sampled ic

values applied during Monte Carlo simulation for the several Delta error surfaces. Mode and 

ErrorControl switches must be appropriately turned on for the effects of the sampled ic

2.5.5 Methods Worksheet: Emission Calculations

values to 

be included in the emissions difference results. These controls affect the calculations in the 

Methods worksheet Parameters and Emission Calculations sections.
256

SwRI Project 14936.12

In the area of rows 6 - 81 of columns CH – CP the brake-specific emissions and 

∆emissions calculations are performed using the variables and parameters generated in the 

Parameters section. Three sets of columns, structured similarly, calculate the full model, 

validation model, time alignment and drift correction for the following methods:

1. Method 1 calculations are applied in columns CH – CJ, 

2. Method 2 calculations are applied in columns CL – CN, and

3. Method 3 calculations are in columns CP.

Columns CH - CJ for Method 1 are typical of the methods where the structure is the same, but 

the formulas are a little different. Column CH performs the PM emission calculations for the 

AVL PEMS, column CI performs for the Horiba PEMS and column CJ for the Sensors PEMS. 

The structure of the three columns is the same. Formulas implemented in the three columns are 

the same, but the equations implementing the formulas apply variables and parameters 

appropriate to the respective PEMS.

As an example of the calculation for PM Method 1 we will examine column CH in detail. 

The full model calculation was accomplished in cells CH48 – CH54. The ideal emissions result 

was brought into the area by equation in CH51. Full model PM emissions (ePM) in g/kw-hr 

were calculated in CH54. Cells CH55 – CH59 are information-only diagnostic aids. The full 

model Method 1 result in CH54 is calculated by the formulas in Figure 1.

m PM

( ) ( ) ( ) ∑

∑

=

=





 



 ⋅ ∆





 



  ∆ 

 



 



 





⋅ = N

i

i i

N

i

PM i

PM Speed rpm T N m t

t

s

mol

n

mol

g m

e g kW hr

1

1

60*1000*3600

* * 2*3.14159*

* *

/



is a flow weighted particulate matter exhaust concentration in g/mol 
257

SwRI Project 14936.12

FIGURE 181. BRAKE-SPECIFIC PM BY METHOD 1

In the formula for the full model mode 2, delta error values sampled from the Delta worksheets 

1, 2, 4 and 5 have been added in m PM

• 20 Delta Exhaust Flow SS

. Similarly, delta error values sampled from Delta 

worksheets 20-23, 25, 27 and 28 have been added to the exhaust flow, delta errors sampled from 

worksheets 29-32 and 34-35 were added to torque, and worksheet 43 deltas were added to speed. 

The ∆t values are equal (1 second) and therefore cancel out of the equation.

The validation model calculation was accomplished in the cells CH79 – CH81. 

Validation model PM emissions (g/kW-hr) was calculated in cell CH81.

Calculations for PM by Method 1 described above for the AVL PEMS in column CH are 

similar for the Horiba and Sensors PEMS by Method 1 in columns CI and CJ, respectively. 

Similar calculations for PM by Method 2 are presented in columns CL – CN, and by Method 3 in 

column CP.

2.6 Worksheet 4: Constants and Equations

The Constants&Eqns tab was strictly a snapshot of equations used in the brake-specific 

emissions calculations. It displayed the equations and constants implemented in spreadsheet 

formulas of the Methods worksheet

2.7 Worksheet 5: SS PM Error Surface

The 7 Delta CO SS worksheet is a typical Delta worksheet. Its functional structure, 

formulas, charts and operation are very similar to the following worksheets:

• 22 Delta Exhaust Flow Pulsation

• 23 Delta Exhaust Flow Swirl

• 30 Delta Torque DOE Testing

• 45 Delta CO2 SS

With minor changes in charts and structure, its function, formulas and operation are also similar 

to the following worksheets:
258

SwRI Project 14936.12

• 1 AVL Delta PM SS

• 1 Horiba Delta PM SS

• 1 Sensors Delta PM SS

• 2 AVL Delta PM Transient

• 2 Horiba Delta PM Transient

• 2 Sensors Delta PM Transient

• 4 AVL Delta PM AtmosPressure

• 5 AVL Delta PM Ambient Temp

• 13 Delta NMHC SS

• 14 Delta NMHC Transient

• 19 Delta Ambient NMHC

• 21 Delta Exhaust Flow Transient

• 25 Delta Exhaust EMI-RFI

• 29 Delta Dynamic Torque

• 31 Delta Torque Warm-up

• 32 Delta Torque Humidity

• 34 Delta Torque Interpolation

• 35 Delta Torque Engine Manuf

• 42 Delta Fuel Engine Manuf

• 43 Delta Dynamic Speed

• 44 Delta Dynamic Fuel Rate

• 46 Delta CO2 Transient

The following provides a brief summary of the 7 Delta CO SS worksheet:

• Rows 1 – 7 contain descriptive information about the error surface implemented in the 

worksheet.

• Rows 8 - 42 present the error surface in columns A – L. Other columns, M – W, on these 

rows generate a lookup table used with an interpolation routine.

• Figures A, B and C follow.

• Rows 76 – 379 calculate the ∆CO SS error values for each row of the reference NTE 

event. These values were returned to the Methods tab Parameters section.

The following paragraphs describe in further detail functions in the 7 Delta CO SS worksheet:

Data from the error surface (rows 13 – 42, columns A – L, in this Delta worksheet) must 

be entered in sorted order (sorted on Lab Nominal column C in ascending order) for proper 

operation of the x-lookup-interpolation function. The three figures chart the error function. 

Figure A, in similar Delta tabs, may plot several data sets versus the x-value, Lab Nominal 
259

SwRI Project 14936.12

(column C). Figure A y-values are CO % Lab Nominal (column C), and may also plot 99th

percentile, 50th percentile (median) and 1st percentile.

Related error surface data are plotted in Figure B. Figure B plots several data sets versus 

the same x-value, Lab Nominal (column C). Figure B y-values are the difference, CO % 

(PEMS) – CO % (lab, nom). The differences plotted may not correspond exactly to the values 

shown in Figure A because of the statistical procedure applied in calculating the differences 

shown in Figure B. Figure B plots the 99th percentile (column I), the 50th percentile (median) 

(column H) and the 1st percentile (column G). In addition to the error surface data, Figure B also 

shows the interpolation line designated ic = xx (column V), and the reference NTE event values 

on the interpolation line (column F rows 80 through end of the reference NTE event versus Lab 

Nominal x-values in column B rows 80 through end of the reference NTE event). When ic = 

+1.414319083, the interpolation line plots on the 99th percentile. When ic = 0, the interpolation 

line plots on the 50th percentile. When ic = -1.414319083, the interpolation line plots on the 1st

percentile. The reference NTE event always plots on the interpolation line, with points at the xvalues in the reference NTE event.

The error surface data were also plotted in the format of Figure C. Again the x-axis was

the same Lab nominal (column C). This time the y-axis data are the ic values. Thus, the 99th

percentile plots at +1.414319083, the 50th percentile plots at 0 and the 1st percentile plots at -

1.414319083. The interpolation line plots at the value of ic

Now consider inner rows 13 – 41 in the look-up table in columns T – W. Column T is a 

repetition of the x-value from column C. Column U calculates a row-to-row ∆ for the x-values 

in column T for use in interpolation. Column V computes the interpolation line linearly 

interpolated according to the value of i

, and the reference NTE event plots 

on the interpolation line at the x-values in the reference NTE event. If appropriate value labels 

were displayed in Figure C, the values would represent the error surface plotted on a z-axis 

above the two-dimensional x-y plane. These error surface values are displayed graphically in 

Figure B.

c between the median and the 99th percentile if ic > 0 (on 

median if ic = 0 and on 99th percentile if ic = +1.414319083); and between the median and the 1st

percentile if ic < 0 (on median if ic = 0 and on 1st percentile if ic = -1.414319083). Only one ic

value (from cell E80) is applied in this calculation of the interpolation line. The Microsoft Excel 

vertical lookup function VLOOKUP is applied to the table in rows 12 – 42 in columns T – W. 

This is done in rows 80 and down in column F. Because of the way the VLOOKUP function 

operates, the first row cells T12 and V12, and the last row cell W42 (all three cells distinguished 

by darker line borders) contain formulas or values different from the formulas of the inner rows. 

The formula in cell T12 assures that the lookup function can always find an x-value in its table. 

The formula in V12 and the value in W12 assure that the interpolation in cells F80 to the end of 

the reference NTE event data returns the nearest ∆CO SS value on the interpolation line if the xvalue is outside the range of the error surface lab nominal values.
260

SwRI Project 14936.12

Before going to the interpolation accomplished in F80 and down, consider briefly the 

formulation on rows 12 – 43 in columns O – R. This formulation considers one x-value from the 

reference NTE event, the first one, in cell B80 and selects the two adjacent rows in the error 

surface between which to interpolate on the B80 x-value. The result is formed on row 43 in 

these columns and then the “check” cell G80 accomplishes the ic controlled interpolation. This 

provides an alternative calculation check on one row in the reference NTE event.

Now consider the interpolation for each point in the reference NTE event. Column B, 

row 80 and down, brings the lab nominal x-value from the Methods worksheet reference NTE 

event. For this Delta worksheet, that x-value is CO % (lab,nom). The out-of-range flags are 

information-only indicating points in the reference NTE event with x-value out of the range of 

the error surface lab nominal. The ic value for this Delta worksheet was brought into cell E80 

from the Methods worksheet ic area. Each point in the reference NTE event was interpolated 

with the same ic value, but with its own x-value. Recalling that the interpolation line in column 

V was computed with this one ic value, the x-interpolation between the appropriate two adjacent 

rows in the error surface can now be accomplished. This requires using the x-value on each row 

in column B, B80 and down, in the VLOOKUP function, and performing the required 

calculation using the looked-up values and deltas from the look-up table. The calculation is done 

with the formulas in cell F80 and down. The values computed in column F, cell F80 and down 

through the reference NTE event, could be considered elements of a column matrix or vector, 

and are returned to the Methods worksheet Parameters section.

In the Monte Carlo simulation, the Methods worksheet combines this reference NTE 

event result vector from the 7 Delta CO SS worksheet with similar results from other error 

surfaces, calculates ∆emissions by three methods, full model and validation to produce a set of 

14 output values (“Forecasts” in Crystal Ball terminology) described in the Summary worksheet

section. This was done having input ic values (including one ic value for this Delta CO SS) all 

chosen by random sample from the appropriate truncated normal or uniform distribution as 

explained in the Methods worksheet section. Then another sample set of randomly sampled 

values was input (only one ic value coming to this Delta function again). The reference NTE 

event CO SS vector was recomputed with the one new ic

The number of rows in the Delta worksheet reference NTE event area (rows 80 and 

down) should match the number of rows in the reference NTE event applied in the Methods 

worksheet for proper function of Figures B and C. The starter spreadsheet has been set up with 

the range of charted reference NTE event series extending through row 379 in this Delta 

worksheet. The balance of the spreadsheet should calculate correctly when a reference NTE 

value, returned to Methods worksheet 

and another set of 14 output values was produced. This process was repeated many times until a 

statistical convergence criterion, described in Section 2, was satisfied. Typically, 40,000 to 

65,000 sets of input values and 14 output values were produced to satisfy the convergence 

criterion with this Error Model spreadsheet.
261

SwRI Project 14936.12

event is properly entered in the Methods tab and Parameters formulas properly aligned, although 

figures like B and C will not display properly until the last row of the reference NTE event is 

coincident with the end of the range of the charted reference NTE event series. This could be

done manually in each Delta worksheet where needed, however, the macro was designed to 

convert the fully ‘loaded’ starter workbook after the reference NTE event was entered in the 

Methods worksheet. The macro uses the row count in the reference NTE event, aligns formulas 

in the Methods worksheet Parameters area, and eliminates extra rows in the reference NTE event 

area of each appropriate Delta worksheet. Again, the macro will do the operations correctly only 

on a fully ‘loaded’ starter workbook set up with 300 rows of formulas in the Methods worksheet

Parameter area, and in each of the Delta worksheets using the reference NTE event.
262

SwRI Project 14936.12
263

SwRI Project 14936.12

APPENDIX D: STEERING COMMITTEE MEETING MINUTES
264

SwRI Project 14936.12
265

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at CE-CERT

April 23, 24, and 25, 2008

Future Meetings:

1. May 15-16 meeting, SwRI

2. June 12 and 13 at for PM-PEMS and 10 and 11 for EMTC, ACS.

3. July 29 and 30, San Antonio, SwRI

4. August 28, 29, San Antonio, SwRI

On-Road PM PEMS Discussion and Action Items

CO2

1. Use CO

Activities

2 data provided by CE-CERT during the gaseous program for CO2

Test Plan Activities

validation, 

and share the information during the May 15-16 meeting at SwRI.

a. Explain in test plan why Horiba can’t use Method 3 

b. Replace BSFC in the equation of Method 2 with (fuel flow/torque *speed). 

Essentially, remove BSFC error surface

c. Show how (mp bar) for the AVL is calculated

d. For AVL Method 3, use a different error surface for PM.

e. For reference NTEs, use the existing reference NTEs, and PM concentrations to 

produce different concentration from DPF out to threshold levels. 

f. Check section 2.3 with Bob Mason to make sure it is correct

g. Change the example to PM in the test plan and give an appropriate PM 

concentration range per mole basis.
266

SwRI Project 14936.12

h. Reexamine the text to explain the new equations better. No second by second for 

AVL

i. Change Figure 8 in the test plan to reflect the fuel flow inclusion, and remove 

BSFC error.

j. Change Figure 8 to update all errors that are required

k. State the time and date by when the model could be available to the group, 

assuming no last minute changes are required.

General Notes:

1. For AVL, use a 3 to 1 dilution ratio and change the selectable range if needed but not 

the dilution ratio.

2. For the NTE windowing, use the EEPS to determine the windowing sensitivity, assuming 

a maximum of 6 seconds delay at the beginning and end.

3. Measure the CVS dilution air temperature as close as close as possible to the exhaust and 

dilution air mixing point. This may require insulation of SwRI CVS system.

4. Use Whatman Teflon membrane, 2 micrometer for the entire program

5. Use the fine ambient backup screen instead of the coarse or diesel backup screen

6. List all the error surfaces in the test plan 

7. Plan Training and Comissioning of the PM Generator

8. If 32 seconds NTE is not sufficient for the PM-PEMS, extend the length

9. Use 10 % difference as an acceptable difference between SwRI and CE-CERT

1065 Audit (For reference see SwRI presentation)

TSI Flow Meters
267

SwRI Project 14936.12

• Using reference flow meters like the TSI flow meter as a transfer standards is 

okay, if independently checked via a master flow meter

Sensors PPMD

a. Verify that the external and bypass flow TSI flow meter certificates are valid for 

duration of testing

b. Verify that the data published on their individual certificates meet the 1065 

linearity thresholds

Horiba TRPM

• Use total flow and dilution air flow for Horiba and make sure they meet 1065 

linearity verifications.

AVL MSS

• Meeting +/- 3 % per point on dilution ratio is acceptable.

Vibration

Perform a frequency sweep with moderate amplitude and log the real time signals from the 

PEMS using a fixed PM generator level. If changes occur, design a frequency cycle around that 

frequency range to test for delta changes in PM with and without vibration. Obtain approval on 

the frequency cycle selected from the MASC before proceeding. If no changes are observed 

during a frequency sweep, there is not a need to test for vibration.

Use two orientations vertical and horizontal for the PPMD. If there is enough space on the 

vibration table at SwRI, test all PEMS units at once. If not, use them one by one.

EMI
268

SwRI Project 14936.12

Use EMI on and off, and screen the real time response of the instrument at a specific 

concentration level. If a response is observed during EMI on/off switching, the MASC will 

decide on what the next steps are.

Environmental Chamber:

1. Temperature and Humidity cycle will be made available by Matt Spears

2. Use a high velocity fan to blow over the 4 inch inlet section of the PPMD

3. PM Generator Setup and Verification

4. How to do the environmental pressure test?

a) Use three different levels concentration at the 0.02 and 0.03 g/hp-hr 

b) Run a long baseline of 8 hours using three different toggeled PM concentration levels. 

For baseline, toggle the process five times, and use that as a baseline, followed by the 

continuation of the baseline.

c) Run the temperature and humidity test using the same toggeling profile. For temperature, 

humidity, and pressure, start with a baseline with five repeats, and then kick off the 

environmental cycle.

d) A toggeling of zero, mid, and high is on the order of 15 minutes.

There is a possibility of eliminating the zero and add one concentration level, and randomly 

sample from all the deltas generated. We need to talk to Bob Mason about this.

Use just one of each of the PEMS for environmental activities.

Finally, changes were made to the test plan during discussions. Also, SwRI presented that 

attached document on the test plan. More test plan discussions will take place in the next meeting 

on May 15 and 16, at SwRI.
269

SwRI Project 14936.12
270

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI

May 15-16, 2008

Day 1

1. Tim French indicates that there are several issues came up with the PM-PEMS onboard 

testing performed at CE-CERT before the CE-CERT meeting was presented.

2. Kent’s presentation, posted on FTP website:

Some highlights, but see presentation for more details:

a. Problems with valve timing opening and closing consistency and long delays over 

3 seconds.

b. There was one crystal that was very noisy but the software did not reject it, and 

reported data using it.

c. A drop in frequency but an increase in mass deposit. (This is a non-issue)

d. Semtech DS 10[1].09 SP2 b5. This beta J1939 includes filtering and was able to 

include NTEs.

e. Post test J1939 filtering versus real time J1939 filtering. Post test captured 

different start up NTE than the real time filtering one. Essentially, post processing 

software needs to be scrutinized.

f. NTE yield produced by PPMD was low. Some of the valid NTE that was captured 

in PPMD post processing was rejected, after carefully reviewing the data.

g. Others. See presentation.

3. SwRI update:

a. Horiba linearity check was resolved by using the Gilibrator directlty

b. Engine A, B, C, and D 80 steady-state testing was complete

c. Engine A was tested for 40 SS points in sub-NTE runs

d. International engine was also tested

e. All above will be posted on the website after careful review
271

SwRI Project 14936.12

4. Janet and Bob Presentation, please see website.

Bill Martin questioned the idea of excluding the 95th to 100th

5. General Discussion and Action Items:

data twice. One in the error 

surfaces, and the other in the measurement allowance. He was concerned whether such 

practice will lead to truncation of the distribution. Janet will show in the next meeting 

that such practice will not lead to any truncation of data or it will effect the shape of the 

error distribution.

a. Find ways to load up the can bus during the actual testing to simulate real world 

operation by making the can bus communication busy.

b. Keep using the gaseous error surfaces for Method 2 and Method 3 on the PMPEMS.

c. Environmental Chamber:

1. Use a 9 minute cycle for the environmental chamber and go into three 

concentration levels, 3 minutes per level. Sample for a period of 35 seconds 

from each level. 

2. Use five levels of dilution ratio ranging from 6 to 50.

3. Check with Sensors to see if they accept an exhaust flow meter analog input.

For Model output, keep the 95th as the measurement allowance. However, if it did not 

validate consider the potential of using other than the 95th

Day 2:

, if the MASC reaches an 

agreement on that.
272

SwRI Project 14936.12

1. Move forward with the International Engine

2. Communicate via J1939 to load the engine can bus during testing. Use only the J1939 

communication with the heavy-heavy duty diesel engine 

3. Accept the idea that Horiba will use test cell engine speed and load analog output signals

4. The right to remove outliers using good engineering judgement.

5. Starting on June 2nd

6. Spend one week of commissioning before we start.

with commissioning

Plot:

x-axis percent of max torque or fuel rate

y-axis absolute difference over max torque or fuel rate

Matt’s Discussion:

Model Validation Testing at CE-CERT

a. Number of engines/vehicles

b. Number of bypass conditions, at least two, maybe 3

c. Number of PEMS: at least two of each, highly desirable three

d. Number of NTE events: total 100-200 per PEMS

e. Number of route repeats: 1 to 3

f. Types of NTE events

i. CE-CERT limits

1. Minimum filter loading: 50 micrograms

Gaseous plus PM. But PM set priorities. Sensors goes first.

NTE Route Bypass Repeat
273

SwRI Project 14936.12

Events

Horiba plus 

AVL

30-50 Palm 

Springs

1 One-Run

Sensors 

plus AVL

30-50 Palm 

Springs

1, review One-Run

Horiba plus 

AVL

30-50 San Diego 1 or change One-Run

Sensors 

plus AVL

30-50 San Diego 1 One-Run

Horiba plus 

AVL

30-50 Baker 1 One-Run

Sensors 

plus AVL

30-50 Baker 1 One-Run
274

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting in Madison, Wisconsin

June 12-13, 2008

The meeting started by reviewing and approving the last meeting minutes.

PPMD Commissioning

a. Following the meeting minutes, Imad Khalek gave a presentation on the status of 

the PPMD commissioning at SwRI. The presentation was sent to the MASC, but 

was not posted on the website. 

b. David Booker also gave a presentation on the SwRI commissioning activities. The 

presentation was sent to the MASC, but was not posted on the website at the 

request of Sensors.

c. As a result of these two presentations, the MASC decided to give Sensors a 

chance to fix some of the problems encountered and come back to SwRI for 

additional commissioning during the week of June 16. Also, the MASC requested 

that a conference call should take place on Friday, June 20, and SwRI should give 

a status update on whether or not the Semtech-DS/PPMD issues were fixed to a 

satisfactory level so the program can proceed.

d. The MASC also made the following points on the PPMD:

i. It is up to Sensors to decide on the quartz crystal equilibration time, after a 

crystal goes into a invalid NTE window. 

ii. It is requested that when all crystals are locked out, and one of them 

becomes available during an NTE, the crystal should wait until the current 

NTE terminates, and a new NTE starts before it samples from an NTE. 
275

SwRI Project 14936.12

PM-PEMS Engine Selection

Imad Khalek pointed out to the MASC the fact that the PM-PEMS program is moving 

forward with some deficiencies related to the Horiba system inability to communicate with the 

Navistar engine ECM ISO protocol. It was recommended to the MASC that a heavy-heavy duty 

diesel engine be installed in the test cell first so the Horiba system can communicate with the 

engine ECM using the J1939 protocol. In addition, this will give a chance for Horiba to upgrade 

their system in preparation for the Navistar engine after a heavy heavy-duty diesel engine.

EMA agreed to take a look at the possibility of providing a heavy heavy-duty diesel 

engine in a timely manner. EMA agreed to make a final decision on this issue by the June 20 

conference call.

Horiba Concern

Horiba was concerned about the fact that Sensors brought a new model of the PPMD to be used 

on the PM-PEMS-MA program. They wanted to have a chance to update their system as well. 

The MASC decided to give Horiba a chance to work and upgrade their systems. It was decided 

that SwRI should ship back to Horiba one of their system present at SwRI, and give them until 

July 14 to ship back the system.

Horiba was also given until September 15 to upgrade their system so it can communicate with 

the Navistar ECM protocol using ISO-15765. 

Bill Martin made comments about the difficulty he sees in the Horiba system meeting 1065 

requirements. Matt indicated that EPA would enter an allowance for Horiba specifically, at the 

time of Direct Final Rule (DFR), through the alternate procedure approval.
276

SwRI Project 14936.12

AVL Presentation

Bill Silvis presented results on the MSS, where a compensation algorithm is added to account for 

organic carbon and sulfate. No copy of the presentation was given to SwRI for distribution. Matt 

Spears was not convinced that such compensation will be acceptable for EPA approval due to the 

significant correction required and due to the MSS principle itself. SwRI also had some technical 

reservation about the process due to its technical complexity. Essentially, the problem is not 

trivial and more thorough work and understanding is still needed. 

AVL was encouraged to continue working on this issue and refine it. It is understood that they 

will submit the compensating algorithms prior to the start of steady-state testing. They may also 

submit their compensating algorithms at any time for the MASC to have analyzed by SwRI or 

CE-CERT. 

Rey Agama suggested that time would be the best window of opportunity for AVL to be 

included via DFR through alternate procedure approval. 

SwRI Test Cell

The MASC requested the following:

• SwRI should install all PM-PEMS as close as practically possible to the entry 

of the CVS. This will reduce any particle losses between the point of 

measurement among the PM-PEMS and also relative to the CVS. 

Furthermore, this will minimize the backpressure experienced by the PPMD, 

by shortening the length of exhaust piping present downstream of the PPMD, 

prior to entry to the CVS. The target backpressure is (-1 to +4 kpa)
277

SwRI Project 14936.12

• SwRI should try to maintain temperature of 35°C ± 5°C in the vicinity of the 

PM-PEMS inside the test cell. 

CE-CERT Presentation

Kent Johnson gave a presentation on the temperature distribution on various location of the MEL 

such as behind the CAB, under MEL trail, and under passenger door. Kent’s presentation is 

posted on the website. One of intriguing highlights of the presentation is that the temperature 

profile can reach as high as 60 °C to 90 °C during in-use in the vicinity of the PM-PEMS. This 

triggers the idea of changing the environmental temperature profile that was used during the 

gaseous PEMS program that was based on ambient temperature. Matt spears will be modifying 

the existing environmental chamber temperature profile, taking into account Kent’s finding.

Budget

CE-CERT presented their budget with some options. The overall impression was that the total 

budget for both SwRI and CE-CERT is beyond the funding level available for this program. 

Below are some of the options entertained that will be discussed during the next meeting at SwRI 

starting on July 21st

1. Reduce the scope of work by cutting the number of routes and the number of PEMS used 

by CE-CERT

.

2. Reduce the scope of work at SwRI by reducing the number of engines to be tested from 

two engines to one engine

3. Increase the overall budget by eliminating this year EMA pilot program requirement, and 

add funding to the MA program instead, assuming that the funding will be cost shared 

among EMA, EPA, and CARB 
278

SwRI Project 14936.12

Additional Comments and Action Items:

1. If PPMD and OBS200, both, resulted in negative allowance, e.g. -0.01 to -0.02 g/hp-hr, 

the MA will be zero, essentially one MA for the entire program and for all the PM-PEMS 

used. Under such circumstances, both instruments will be allowed to be used with a zero 

MA.

2. Keep aware of alignment issues. One may want to investigate the brake-specific emission 

value reported by PM-PEMS by shifting the numerator and denominators relative to one 

another and relative to absolute time. This could be done with initial transient test results 

with bypass.

3. Sensors will provide a VI to simulate the exhaust flow rate so we can exercise the 

multiple dilution ratios in the environmental chamber. 

4. SwRI will need to resubmit some budget options of doing the work with one engine 

versus two engines. 
279

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

July 21-23, 2008

July 21

The meeting started at 1:00 PM. It started by reviewing and approving the previous minutes, and 

also by discussing future meeting schedules, which were already sent by EMA as future meeting 

notices. The next meeting is scheduled for August 27 (starts at 2:00 PM), 28, and 29 (ends at 

2:00 PM).

Imad Khalek started a presentation, posted on FTP website, on project status update. He showed 

a comparison between laboratory-based exhaust flow and the 4-inch Sensors exhaust flow that 

was used on the Navistar engine. The slope of the correlation was too high at 1.27. As a result of 

the discussions, the MASC decided on the following with the current 5-inch exhaust flow meters 

used with the Mack engine, which the first engine to be used as a part of the official 

measurement allowance work.

• Check laboratory-based exhaust flow rate with Horiba and Sensors 5-inch exhaust flow 

meters. If a problem is obvious or the slope of the correlation differs by more than +/-

0.05 from a slope of 1, then send the flow meters back to the manufacturers to check on 

the calibration.

After finishing a part of the presentation, Janet Buckingham and Bob Mason showed up for a 

scheduled presentation at 3:00 PM. The presentation is posted on the FTP website and addressed 

the double truncation issue raised by Bill Martin at the 5th and 95th

a. The 95

percentile. The presentation is 

posted on the FTP website. Based on the presented work, the following was agreed upon:

th

b. The 5

percentile is still desired for the measurement allowance

th and 95th are still acceptable to bound the validation range.
280

SwRI Project 14936.12

c. There was still a remaining unresolved issue about where you assign the -1, 0,1 on 

the error surfaces for the delta change between lab and PEMS. It was decided that 

this issue should be addressed during the last day of the meeting, but that was 

never brought up again. There was a proposal by Bill Martin to fit both sides of 

the error distribution independently using a normal distribution fit. The median of 

such distribution will be the median based on previous practice along with 95th for 

one side and 5th

July 22

A significant part of the day was allocated for budget discussion. The different budget scenarios 

are posted in the FTP website. Below are some of the highlights:

For budget cutting purposes, one engine will be used for the measurement allowance at SwRI. In 

addition, work will be performed with one bypass setting that gives brake specific PM emissions 

levels between the two thresholds of 0.02 and 0.03 g/hp-hr. For CE-CERT, choose the PEMS 

with the lowest positive measurement allowance. If one PEMS clearly shows a lower 

measurement allowance than the rest, and both are positive, pick the one with the low 

measurement allowance for the rest of the validation testing.

If one allowance is positive by one PEMS, and if the other one is negative, then, in principle, 

choose the positive allowance if it slightly positive. (no clear cut agreement yet).

After the budget discussion, Tim French from EMA walked the group (via phone) through the 

EMA proposal, posted on FTP website, to substitute year 1 pilot program and to also provide 

some supplemental funding on the order of $200 K to inject more funding into the measurement 

allowance program. The overall EMA budget funding was projected to be on the order of $500 K 

to $700 K. Most or all the proposed activities will be performed in-use by hiring a third party 

that is not part of EMA and does not belong to one of the engine manufacturer to conduct the 

testing. As a result of the EMA proposal the following will take place:

for the other side. When each side of the distribution is fitted 

using the above boundary conditions, one can then expand the data picked from 

the error surfaces to cover 1 % to 99 % of the data or even 0.1 % to 99.9 %. No 

decision has been made on this issue yet. This topic will require more discussion 

during the next meeting.

• Matt Spears will speak with EPA upper-management to consider the EMA proposal in 

the context of the one year pilot program.

• EPA will also make a final determination on whether to use the shortened version of the 

measurement allowance program. E.g. one versus two engines, by-pass, no-bypass, etc...
281

SwRI Project 14936.12

After the budget discussions, Imad Khalek continued his presentation that was started on the first 

day. The work showed the different torque and fuel errors surfaces determined between the 

laboratory and the engine ECM public broadcast. The data covered a total of five engines (four 

heavy-heavy and one light-heavy) that included a DDC Series 60, CAT C13, Cummins ISM, 

Mack MP7, and Navistar 6.4 liter engine. 

Also the work covered sub-NTE fuel flow errors. Based the sub-NTE results, it was decided that 

for forced NTEs, if the engine operation falls below NTE, use the last BSFC value observed 

within the NTE. Use that only for Method 2 calculations.

After this presentation, Imad Khalek refreshed the memory of the group by giving a status update 

presentation, posted on the FTP website, based on the last commissioning work done at SwRI. 

The presentation was posted on the FTP website. After that, Kent Johnson from CE-CERT 

shared some observations about the Sensors data produced by SwRI during commissioning. 

There was no conflict between SwRI and CE-CERT reporting on the results. Kent’s Word 

document that was shared with the group will be posted at the FTP website when it becomes 

available to SwRI.

July 23, 2008

The focus of this day was on the Test Plan, particularly the environmental testing. However, at 

the beginning of this day, the idea brought by SwRI earlier of adding short NTE windows for the 

laboratory transient testing, along with some low or medium idle operation prior to the NTE 

transient cycle was discussed.

• It was decided that Short NTE windows should be added to the thirty 32-seconds NTE 

cycle, in the inter time. A total of 10 short NTEs will be added as follows:

1. 4 five seconds NTEs, with two back to back within an inter-time

2. 3 10 seconds NTEs

3. 3 18 seconds NTEs
282

SwRI Project 14936.12

Position two five seconds NTEs back to back in the inter-NTEs. For Horiba, they 

need about 2 to 3 seconds to exit an NTE. Thus, place the short NTEs at least five 

seconds after the end of a valid NTE to avoid continuous sampling with Horiba.

• On the conditioning prior to the start of a transient NTE, start with five to 10 minutes at 

medium idle time before the NTE transient cycle to observe if there is an effect on the 

laboratory PM emissions results. If there is and effect, then propose an idle time or 

something similar prior to starting the NTE transient cycle. (Note that before each NTE 

transient cycle, the plan is to force-regenerate the DPF first, medium speed idle will then 

be added after the forced regeneration).

EMI/RFI/Shock &Vibration

1. For EMI/RFI, expose the PM PEMS to EMI/RFI using HEPA filtered air, similar to the 

gaseous program. Then decide with the MASC after reviewing the results, what is the 

next step. As an option, we could use the PM generator to perform this work. One PEMS 

from each manufacturer will be used for these activities.

2. Vibration and Orientation (non-road only)

a. On the orientation, ask PEMS manufacturers on the worst orientation scenario 

postion. Use such orientation at 45o

b. Perform a frequency sweep, with a very moderate amplitude, then share with 

MASC to decide on how to go forward. Eric should propose the sweep frequency, 

amplitude and duration. Do two repeats. One of each instrument will be used. 

This is only for nonroad. Outside the scope of the program.

with the appropriate orientation, and survey 

the worst case scenario for all gaseous and for PPMD, and TRPM. 

3. As for shock, ask Eric about a recommended on-highway profile for shock.

Temperature/Humidity chamber and Pressure Chamber

For both temperature and humidity and pressure chamber work, use three PM concentration 

levels around the two threshold levels such as 4000, 5000, and 6000 μg/m3. Use a total of four 

dilution ratios of 6, 12, 20 and 30. At each dilution ratio, stabilize for 4.5 minutes at each 

concentration level and measure for 30 seconds before you go to the next concentration level to 

do the same. E.g. 
283

SwRI Project 14936.12

1. Set the dilution ratio to 6, stabilize at 4000 μg/m3 for a 4.5 minutes

2. Measure PM using PEMS for 30 seconds

3. Move to next concentration level of 5,000 μg/m3 and stabilize for 4.5 minutes

4. Measure PM using PEMS for 30 seconds

5. Move to the next concentration level of 6,000 μg/m3

6. Measure PM using PEMS for 30 seconds

7. Repeat 1 through 6 at different dilution ratio

A spreadsheet on PM concentration and loading was originally made by Matt Spears during the 

meeting and was very slightly modified by Imad is posted to the FTP website

As a result of the proposed scenario above, Sensors PPMD will require a secondary dilution 

using MPS2 to perform an 8 hours of activities using the above scenarios. In case of Horiba, 

either we need to allow higher dilution ratio than those adheres to 1065 or we need to allow more 

loading on the filter past a 0.4 mg. More discussions on this will need to take place during the 

next meeting.

The PM cycle to run by the PM generator contains five 15 minutes cycle to be used as a baseline. 

E.g. Run the first five 15 minutes at normal temperature, use those as a baseline line, then 

proceed for the rest of the day to capture an eight hours of similar repeats. Do the same thing 

prior to starting the Temperature/Humidity profile and well as the Pressure Chamber profile. One 

of the remaining issues that has not been resolved around this topic is how do we capture the 

baseline information with the Horiba system without the need to change the filter. More 

discussion on this subject is needed during the next meeting.

New Temperature Profile

Take the mean of CE-CERT Cab minus ambient and add it to original temperature profile, and 

solve for new temperature and humidity profile that maintains derived from the real world data.

Perform a Fourier transform on the CE-CERT Cab data, eliminate frequency content that are 

similar to the base ambient profile, and use a magic synthesizer to superimpose the frequency on 

top of the new temperature profile. Matt Spears is assigned to do this task. 
284

SwRI Project 14936.12

Pressure Chamber

We resolve the logistical issue of the pressure chamber work by placing the PM generator inside 

the chamber. Matt will need to send me some dimension on the PM generator to see if it fits in 

the Chamber. Other gases as well as 30 amp circuits should all be accommodated.

For Horiba, heat trace the segment of the transition to 250 oC, similar to the PM generator outlet 

temperature for both the Horiba and AVL. 

Use each instrument separately for these tests.
285

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

August 27,28, & 29, 2008

Meetings

October: 9,10-at EPA

Nov: 12,13, and 14-at SwRI

Dec.: 10,11,12-at SwRI

Discussion Points and Decisions Made

• For the error distribution choose the 1st and 99th at -1 and 1, based on a normal 

distribution fitting to extrapolate down to the 1st and 99th

• Recheck the 4 inch flowmeter calibration that was on the International engine

. For the height of the median, if 

there is a discontinuity, choose the average or randomly pick one or the other.

• For Horiba, filter loading is allowed to go beyond 400 microgram up to 700 microgram 

or beyond as long as the flow is controlled.

• For Horiba, do the five cycle baseline at the beginning for humidity and temperature and 

do it at the end for the pressure. This will require changing the filter. For the pressure, if 

it can be done within an hour than do it at the beginning, otherwise, do it at the end.

• Which of the three units are acceptable to be used in the pilot program:

 PPMD is approved

 Horiba might be approved
286

SwRI Project 14936.12

 AVL will have to make the case at EPA to see if approved

 EMA would like to know what instrument would be acceptable for pilot 

one program

 Can EMA combine pilot efforts within companies

 EMA will fund the additional funding required that will be required to do 

the intermediate testing that involves:

• One bypass setting, one engine at SwRI

• Three PEMS, one manufacturer with CE-CERT

• We should move forward on RMI, RFI, and vibration sweep, as soon as we start the 

official testing 

• Due to spikes, we may revisit the post processing of the Horiba results, especially during 

the NTE transient test. As of right now, Horiba first correlates the entire EAD signal 

(including spike) with the filter weight (not including spikes), then apply the relationship 

for the NTE window portion of the cycle.

SwRI Presentation and Action Items

During the meeting, SwRI gave a status update via presentation, see enclosed 

presentation (Update 6), on the PPMD, MSS, and TrPM. SwRI, also discussed the bypass tuning 

for using 10 NTE transient cycle, 80 points steady-state using MSS, and projected filter mass 

concentration, using a relationship between the filter and the MSS. In addition, the bypass 

mixing was shown qualitatively. Furthermore, the transient concentration trace with the MSS 

was presented for the 10 NTE transient cycle.

As a result of the work reported several action items were born:

• Make sure that the filter-based steady-state projected concentration is consistent with the 

filter-based measured for the six steady-state points selected. Thus, perform an 

experiment to determine the filter-based measured concentration for the six points 

selected and share the results with MASC

• Instead of showing a qualitative results on the mixing, provide some quantitative 

assessment such as a T-test.

• Based on the high spikes observed with the MSS, check if the PPMD results in a low bias 

due to the fact that it may be missing the early spike. For these experiments, pick the 

transient NTE window with the highest spike and create a cycle that consists of 10 

repeats of that same window. During this exercise, use only three working crystals and 
287

SwRI Project 14936.12

vary the PPMD trigger into the NTE with time advancement of 1,2 and 3 seconds, time 

delay of 1 second, no time delay, and ECM trigger

The above work and the problems encountered were presented to the MASC via two conference 

calls that were done on September 12 and September 30, 2008. The presentations are also 

enclosed as Update 7, and Update 8.
288

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at EPA, Ann Arbor

October 9 and 10, 2008

SwRI gave a status update on the progress of testing. See enclosed presentation.

There was a concern about the predicted lasting time of the PPMD in the field, which was 

predicted to be on the order of 1 hour, using an average NTE threshold of 0.02 g/hp-hr. There 

was a desire to extend the running time of the PPMD to at least 2 hours. This is indeed met by 

the instrument if one takes into account overall collection efficiency of the instrument. In the 

example given, it was assumed that the collection efficiency was a 100 %, where in reality it was 

on the order of 50 %. Thus the one hour of lasting time reported is in reality two hours.

A decision was made to move forward with testing using only the MPS1, as shown below, after 

final commissioning of the PPMD by Sensors. Actually, David Booker from Sensors flew to 

SwRI late on October 9 to be at SwRI on October 10. 

1. Test on MPS1 only, as its currently configured at one microgram.

2. Okay to use external trigger for steady-state testing

3. For steady-state testing:

a. Target 100 microgram on the filter

b. 50 microgram for TRPM

c. 0.66 microgram for PPMD

Do not clean crystals until it is apparent that the next run will likely overload the filter.

The second day of the meeting was spent at Sensors. Matt Spears gave a presentation on the PM 

generator. Later, he showed the PM generator setup, and explained to the group the various 

elements of the PM generator and the equipment used. A copy of Matt’s presentation is enclosed. 

Also, a copy of Matt’s note is shown below, particularly to 1065 PEMS changes:
289

SwRI Project 14936.12

Matt’s Note:

1. Review minutes from last meeting

2. Discuss 1065 changes required for PM PEMS

3. Update from SwRI on recent activities

4. Update from PEMS manufacturers regarding recent phone conference

5. Friday afternoon @ Sensors

a. SUN conference presentation

b. PM Generator

6. Discuss 1065 changes required for **PEMS field testing only**

a. Dilution Air

i. Temperature control

1. 1065 = 25±5 °C

2. PM PEMS= no direct feedback control from a dilution air 

temperature measurement required-- use good engineering 

judgment; if directly and actively controlled then target 25 °C .

ii. You may use a fixed molar mass of the diluted exhaust mixture for all 

PEMS field testing, as determined by engineering analysis.

b. “Filter”

i. Media 

1. PTFE membrane or TX-40

ii. Face velocity

1. 1065= target near 100 cm/s actual, unless overloading

2. PM PEMS

a. Flow-through media: (10 to 100) cm/s actual, which can be 

verified by engineering analysis

b. Non-flow through media: no specification

iii. Temperature

1. 1065 is 47±5 °C

2. PM PEMS target (42 to 52) at all times, with a minimum tolerance 

of 32 °C and a maximum tolerance of 62 °C, where the tolerances 

apply only during filter sampling.

iv. Conditions during mass determination

1. 1065=see subpart B

2. PM PEMS

a. If mass is not determined in-situ—i.e. within the PEMS—

then the sample collection media must be pre and post 

analyzed according to 1065.190x.

b. If mass is determined in-situ, follow .195.

c. In subpart J, have no requirement to hold to dewpoint 

specs for in-situ analyzers.
290

SwRI Project 14936.12

c. Absolute reference for inertial balance

i. Current status: QCM OEM stated specs are assumed.

ii. For 1065 measurement allowance audit we had Sensors verify frequency 

measurement circuit.

iii. No immediate solution available

d. Cleanup 915 table for inertial batch PM analyzers: no freq, or rise/fall time specs. 

Recommend a process for determining noise, accuracy, and repeatability

e. 1065 Subpart J needs to state that field testing applies at any ambient temperature, 

pressure and humidity, unless otherwise specified in the standard setting part.

f. State that EPA approves of electrostatic deposition technique for PM collection. 

Must meet 95% collection efficiency, as stated by the manufacturer.

g. Overall PEMS test requirement should be reread and edited to be applicable to 

batch analyzers. For example describe how to use a combination of steady-state 

and transient test modes to determine accuracy and repeatability separately; like 

what we’re doing in the measurement allowance.

h. 1065.308-09: also required for continuous PM analyzers—read and edit 

accordingly

i. 1065 clarify that options after 400 ug loading are optional

j. Clarify whether or not ambient air may be used for zero air for PEMS, including 

for hangup check.

k. Drift: allow any drift that doesn’t affect your ability to demonstrate compliance 

with the applicable standard.

7. Timetable

a. Next meetings

8. Test Plan-documentation reflecting the latest agreements

a. Validation—get from CE-CERT

b. Modeling

9. For PEMS testing set dilution ratio based on manufacturers literature regarding maximum 

exhaust flow. You may also use other manufacturer information to perform an 

engineering analysis to estimate the maximum.

10. There will be no dilution ratio verification.
291

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

November 12 and 14, 2008

Flow Alignment in PEMS

In case of continuous sampling, use a step function to measure the delay from the probe to the 

instrument. 

In case of batch sampling, use the geometry to account for time delay from the probe to the batch 

sampler

The above is done to align the flow or to account for time delay.

Leave alone any time alignment between exhaust flow and ECM torque and speed. Use the ECM 

torque and speed to determine the integral over the NTE.

Loss Correction

a. The principle of PM loss corrections for PM PEMS is agreed upon by the steering 

committee

b. EMA desires a legal construct in 1065 for allowing the use of PM loss corrections

i. Open up 1065.295 to allow more types of compensating algorithms, based 

upon other variables

ii. Utilize Subpart J overall approval test to validate entire PM PEMS, 

including its loss corrections.
292

SwRI Project 14936.12

The PEMS manufacturers will be allowed to use compensation algorithms. 

• Horiba decided to use no particle loss algorithm.

• AVL presented a loss algorithm to correct for thermophoresis. The loss correction is 

already defined and will be implemented via a post processor provided by AVL

• Sensors plans to correct for particle loss and will share the process with MASC during 

the next meeting

As of today, no filter data can be shared with Sensors unless the loss correction is shared with the 

MASC.

Test Matrix, DPF out

Full day no QCM cleaning or filter changing

Storage, high speed, light load

Release at peak torque, run for 20 minutes

Store again at high speed, light load

Release at rated power

Do another storage at low idle, equivalent 

Cycle the PEMS every one minute to sample into NTE.

Cycle the PEMS every 32 seconds
293

SwRI Project 14936.12

5 second dwell time in between the NTEs

PEMS Daily Checks Tolerance

Slope of 0.96 is acceptable by Sensors

For Horiba, 3% on filter flow, and 3 percent at a dilution ratio of 5, and 5 percent at a dilution 

ratio of 15, and Mike will reconfirm.

We will get a feedback from AVL on the accepted tolerance for a dilution ratio of 5.f

Milestones

Milestone for the Model dry run

PM generator milestone for commissioning, week of January 5 

Milestone for the fuel flow error surface delivery by the engine manufacturers (EMA will target 

the end of February for these data to be available. The eight or nine engines that are available 

now will be used for the dry run).

Fixed date for the delivery of environmental chamber kit by the PEMS manufacturer. 

Ship it on the 19th of January by Sensors and Horiba. 

We will schedule EMI, RFI, on the 26th

Wednesday

of January, and shock and vibration on the following 

week.

Matt Spears’ Note:
294

SwRI Project 14936.12

1. Upcoming meetings, December, January, and March all at SwRI

a. December 10-12 at SwRI (10th

b. January 28

2pm start, 12 2pm close)

th – 30th (28th 2pm start, 30th

c. March 18

2pm close)

th – 20th (18th 2pm start, 20th

2. Review October meeting minutes

2pm close)

3. EPA / SwRI / CE-CERT PM filter round-robin

a. Initial results

4. Temperature-Humidity test cycle

a. EPA cycle

b. SwRI addition of CE-CERT frequency content

5. Status and progress at SwRI since last meeting

a. Update on mission time projections

i. Relook at projections to see if it is possible to collect 20 failed NTE events 

( 0.02 g/bhp-hr) in one mission

b. Test plan review

c. Decisions for MASC

6. Budget update

a. Arrangements for EPA/EMA/ARB

Thursday

7. PEMS PM loss algorithms

a. AVL

b. Sensors

c. Horiba

8. Test plan development

a. Inclination discussion

9. Continue if necessary: Status and progress at SwRI since last meeting

a. Decisions for MASC

Friday

10. Continue PM PEMS related Part 1065 changes

11. Other meetings during our November meetings

a. Wednesday

i. Wrap-up by 5:30pm due to another room reservation

b. Thursday

i. 12:00pm EMA, Rey Agama – break for lunch at 12:00pm

ii. 1:00pm – 2:00pm, Shirish Shimpi, continue meeting

c. Friday

i. Matt Spears 10:00am to 10:45am, continue meeting

12. Loss corrections resolutions

a. The principle of PM loss corrections for PM PEMS is agreed upon by the steering 

committee, including EMA, EPA, ARB.

b. EMA desires a legal construct in 1065 for allowing the use of PM loss corrections
295

SwRI Project 14936.12

i. Open up 1065.295 to allow more types of compensating algorithms, based 

upon other variables.

ii. EPA agrees that PM loss corrections will not be applied to certification 

testing. If in the future EPA desires to apply PM loss corrections to 

certification testing, such a provision would be proposed as part of a 

notice of proposed rulemaking because EPA acknowledges that such a 

change would cause a change in the stringency of the certification 

standard.

iii. EPA will be the approving body with respect to PM PEMS PM loss 

correction.

1. May make case-by-case approvals, based upon specific PM PEMS 

manufacturer circumstances, such as, but not limited to submitted, 

models, theory, validation data, or even simply the magnitude of 

the correction.

2. May utilize Subpart J overall approval test to validate entire PM 

PEMS, including its loss corrections.

3. May develop (with consultation with EMA) other procedures for 

codification within Part 1065.

13. Sensors will provide an advance copy of Sensors’ December presentation, requesting 

EMA question consolidation ahead of meeting.
296

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

December 10-12, 2008

SwRI presented the steady-state testing results, and the storage and release and regeneration 

results. The presentations were posted on the FTP site.

Sensors presented their approach to particle loss correction. A presentation is available at the 

FTP site.

Craig Kazmierczak presented some in-use PM-PEMS work that was done on one of DDC trucks 

using the PM-PEMS equipment. The work was done by Sensors and presented by Craig.

The MASC requested that the reference crystal be used during SwRI laboratory activities, if that 

is to be used on the road.

The testing done by SwRI so far is acceptable. For the particular PPMD used, disable the 

reference crystal, and use a working crystal to be a reference. Essentially, operate with six 

working crystals, and use one for reference.

Sensors intends to use the reference crystal for correction, but they are going to use a logic to 

decide whether or not it will be used in the post processor.

The post processor should be available before January 23, 2009. The post processor should 

include any loss correction intended or any reference crystal correction model, remove all Part 

1065 excursions, and all other miscellaneous items that will make a measurement invalid.
297

SwRI Project 14936.12

In the current AVL post processor, thermophoretic loss was capped at 25 %. AVL wants to 

change that to include the full range of the model, and remove the 25 % cap. Both corrected and 

uncorrected data will be provided in the output of the post processor.

We agreed to use the paired analysis for steady-state and for model validation.

We agreed to use 5 % COV at a 1 sigma standard deviation for the CVS at 100 microgram filter 

loading.

The above subject was tabled for further discussion on how to subtract the CVS error 

contribution. Bill Martin will send his proposal to Bob and Imad.

How to account for a regeneration in the field:

NTE event > 30 seconds to be valid

1-Discrete

2-Triggered by ECM

3-The regen is defined between two regen flags

4-If the regen occurs during an NTE event, and the length between two consecutive NTE flag x 2 

is shorter than the NTE event, then the regeneration is counted

5-If the regen occurs before an NTE event, and end outside the NTE event, the NTE will count. 

Tim French mentioned that the subject of regeneration and its inclusion in NTE need to be 

discussed in a different forum.
298

SwRI Project 14936.12

Blow-By and how it will be computed for the measurement allowance. Next meeting

As of now, the results reported for storage and release and for regeneration will not impact the 

MA program.

Matt will share the filter results during the next meeting.
299

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

January 28-30, 2009

The last minutes was reviewed and approved. The scheduling for future meetings and 

conference calls was as follows:

Conference Call, Thursday, Feb. 26, 2009, 9:00 am Central, length will be decided on Friday.

Next MA-SC, April 1st(2:00 to 6:00), 2nd (9:00 to 5:00) and 3rd (8:00 to 2:00), San Antonio.

Week of the 18th

• Some of the in-use testing performed by Ce-Cert will include regeneration events. No 

action will be taken on modifying the test plan at SwRI. The test plan will remain the 

same and it will not include active regeneration.

of May for EMTC and MASC (May 20, 21, 22, same as the April meeting)

• Bill Martin explained his proposal on the CVS variability addition to SS delta data. 

Bill Martin presented the analysis on how to assign the CVS variability into the delta 

between PM-PEMS and CVS in order to shrink the 95th and 5th A copy of his writeup is on the FTP website

• SwRI presented work on the progress made. A copy of the presentations is posted on 

the FTP website

• Rey Agama presented an argument about using standard deviation instead of MAD 

for the data analysis. Matt Spears suggested that the MAD should be used, and if no 
300

SwRI Project 14936.12

validation was observed at the end of the program, other possibilities can be 

considered. The group agreed to move forward with this approach. Rey asked that we 

apply a normality test on the data, and he will consult with Bob Mason on that. This 

is in relation to applying a MAD or SD for the model. Bob Mason will present some 

material on the normality criteria during the next meeting.

• The contribution of blow-by will be a constant based on the data presented from the 

four ACES engines at 0.00042 g/hp-hr. If the crankcase is vented to the atmosphere, 

this value will be added to every NTE emissions value. If the crankcase is closed 

loop, the blow-by contribution of 0.00042 g/hp-hr will not be added to the NTE 

emission values.

• Horiba was allowed to fix a bug in their system in relation to delays between engine 

and OBS and OBS and TRPM

• Horiba was allowed to make modifications on the ion trap voltage of the EAD for the 

third TRPM unit. 

• The engine manufacturers agreed to submit the ECM/Lab Torque and Fuel Flow by 

March 31st, 2009. The data will not be linked to a particular manufacturer.

• The instrument manufacturers needs to submit the latest post processor by Friday, 

March 6, 2009

• The MASC agreed that the AVL measurement allowance will be performed based on 

data measured plus thermophoretic correction. The MASC agrees to also see data 

from the AVL MSS total PM prediction on a non-interference basis from the core of 

the measurement allowance program.

• The MASC was updated on the PM generator via a laboratory tour. The PM generator 

is currently set up at SwRI Particle Laboratory. 

Below is the unedited Matt Spears’ minutes:
301

SwRI Project 14936.12

1. January meeting agenda

a. Next meetings?

b. EPA participation/management in MASC

c. Review of Meeting minutes

d. Regen in NTE discussion

e. Bill Martin’s paired testing proposal

f. SwRI data

i. Transient

ii. 2nd

iii. Lessons learned, problems?

set of PEMS

iv. Analysis of MSS with sulfate and HC corrections

v. ECM vs test cell torque and BSFC error surface update

1. SwRI approach

g. PM Generator update

h. Environmental chamber update

i. PEMS mfr readiness

ii. SwRI readiness & schedule

iii. PM Generator readiness

i. Next face-to-face meeting: April 1 afternoon, to April 3rd

j. May 13-15, placeholder-SwRI

afternoon—SwRI

k. Bypass sizing for model validation work

i. DDC engine

l. Filter results

m. Friday Schedule

i. 9:00am start

ii. Review test plan timeline

iii. Environmental chamber update

1. PEMS mfr readiness

2. SwRI readiness & schedule

3. PM Generator readiness

iv. AVL data with corrections

v. Horiba general topic

vi. Finalize agenda for February 26th meeting—finalized, finalize time. 9am 

central. 4 hrs. 12:00pm CST – 4:00pm CST (1pm-5pm EST) Shirish 

travelling to NRMM in Ispra Feb 26th

1. Agenda

a. Make this a LiveMeeting, this is ok w SwRI

b. Update on PEMS manufacturer post processors: 1-hr

c. Timeline update—SwRI: test cell and environmental 

chamber testing: 30 min

d. CE-CERT update on ability to come to SwRI for 

correlation testing & progress on bypass: 30 min

e. SwRI data
302

SwRI Project 14936.12

i. PEMS set number 2: steady-state & perhaps 

transient results—summary only: anything 

remarkably different than the 1st

ii. Any new issues or difficulties: 30 min

set of PM PEMS: 

1-hr

f.

vii. PM Generator / nanoparticle lab tour

viii. Engine manufacturers to submit ECM/test cell torque/fuel rate data 

by March 31st

ix. Rudy’s for lunch

meeting
303

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

April 1-3, 2009

PM-PEMS Meeting: April 1-3, 2009.

Wednesday, 2:00-6:00 PM

• Meeting Minutes from previous meeting were reviewed

• Practice measurement allowance on incomplete set of data was presented by Janet, see 

presentation on website

• The viability of using Method 3 was discussed. There was a general agreement that this 

method needs to be dropped out mainly because the lack of information on time 

alignment between ECM fuel flow and gas-based fuel flow, but such decision needs to be 

reconfirmed during the next meeting.

Thursday, 9:00-5:00 PM

• The normality test requirement was discussed by Bob Mason, see presentation on 

website. The decision for now is not to assume normal distribution and use the MAD 

instead of SD. If the MAD failed to produce a validated measurement allowance, the SD 

will be revisited.

• David Booker from Sensors presented the features of the new PPMD post processor, see 

presentation on website

• Mike Akard from Horiba gave an update on the status of Horiba’s post processor, see 

presentation on website

• SwRI gave a presentation on the following (see presentation on FTP website):

• Status update and project progress

• SS data from all the PM-PEMS

• Transient data with and without engine drift from all the PM-PEMS

• OC/EC data for SS
304

SwRI Project 14936.12

• EEPS data for SS

• Some MSS corrected data for sulfate and HC

• Reference NTE using method 3 gave different values than Method 1 and 2, and 

decision will need to be made on Method 3 during the next meeting.

Friday, 8:00-2:00

• Vibration for offroad was discussed. SwRI plans to do a vibration sweep similar to onhighway but while the PEMS sitting at the vibration table at 45 degree angle. SwRI will 

present a cost estimate for this additional activities when the vibration activities start.

• SwRI showed the fuel and torque data submitted by all engine manufacturers that 

included Caterpillar, Cummins, Detroit Diesel, Navistar, and Volvo Powertrain. SwRI 

will present all data together during the next meeting. 

Some of the decisions/action items made during this meeting were as follows:

• A decision on whether or not Method 3 should be used (AVL only) needs to be made

• Reference NTEs will be tweaked so the main distribution of events will be between 

0.015 g/hp-hr and 0.035 g/hp-hr

• SS error surfaces will be presented during the next meeting

• Transient data was drift corrected using the CVS, but the MASC desired to look at the 

integrated AVL data as compared with the CVS to look at the possibility of using real 

time AVL data for drift correction. The integrated AVL for the NTE cycles will be 

compared with the integrated CVS during the next meeting, before making a final 

okay on the CVS drift correction method

• EPA was to provide some information on temperatures experienced during off-road 

in-use activities so it can be incorporated with temperature and humidity profile 

during environmental testing. EPA was to propose a final temperature and humidity 

profile for the program, after incorporating the CE-CERT and off-road data

• Off-road vibration tests will be added at 45 degree angle

• Fuel and torque errors using the engine manufacturers’ submitted fuel and torque 

needs to be presented during the next meeting 
305

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

May 20-22, 2009

PM-PEMS Meeting, May 20-22, Meeting Agenda

1. Review of last meeting minutes

2. Next Meeting Schedule

3. Overall Project Status Update

4. CE-CERT and SwRI Correlation

5. Update on EMI and RFI Testing 

6. Update on Environmental Chamber Testing

7. SS Error Surfaces and Adjusted Reference NTE events

8. Other agenda items and questions

• The last meeting minutes were reviewed

• The next meeting schedule was set to July 15-17 in Indianapolis

• The overall project status we presented. See presentation by SwRI

• CE-CERT and SwRI Correlation was presented. See presentation by SwRI

• EMI and RFI testing results were presented. See presentation by SwRI. No error 

surfaces will be generated. The problems mainly affected instrument functions.

i. Horiba will investigate the issue related to Bulk Current Injection effect on 

the Horiba exhaust flow 

ii. Sensors will demonstrate a fix to the exhaust flow problem during the next 

meeting. Sensors will conduct their own testing if necessary.

• SS error surfaces were presented. Sensors data were not fully analyzed because 

the new post processor was not provided to SwRI. Sensors promised to provide a 

new post processor resolving the issues identified by SwRI during the last 

meeting to increase data yield. Sensors later provided the new post processor, and 

the SS error surfaces were presented during a conference call that took place on 

June 29, 2009. See SwRI presentation for the June 29 conference call.
306

SwRI Project 14936.12

• The transient errors were also discussed, and the approach for the transient error 

surface will be presented during the next meeting in Indianapolis, along with a 

final recommendation

• Sensitivity on fuel flow and CO2

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, Indianapolis

July 14-17, 2009

flow for Method 2 and Method 3 will be 

discussed during the next meeting in Indianapolis

1. Last Minute Review

2. SwRI Presentation

a. Project Update

b. SS and Transient Error Surfaces treatment and results

c. Environmental Testing (Atmospheric Pressure, Temperature and Humidity) 

Results and Discussion

i. Three approaches were presented on how to treat the error surfaces for the 

environmental testing. 

ii. Approach 3 was used. The steering committee agreed with using approach 

3

iii. No error surfaces were obtained for Horiba and Sensors for environmental 

testing

iv. Error surfaces for AVL, but anything below the lowest MAD, set to a 

constant equal to the lowest MAD, and anything above the highest MAD 

set to a constant equals to the highest MAD

d. For Method 3, and for the reference NTEs, we agreed that the gas based fuel flow 

will be advanced in order to match the ECM fuel flow. 

3. Rules on Measurement Allowance:

a. Pick the positive measurement allowance that is closet to zero based on the 

Horiba’s and Sensors’ PEMS.

b. If both Horiba’s and Sensors’ PEMS have a negative measurement allowance, 

pick the one that is closest to zero.

4. Preliminary conference call scheduled for August 20, 2009. We will confirm it tomorrow.

At the August’s conference call a decision on the measurement allowance will be made 

and an instrument will be selected to go to CE-CERT. After that, SwRI will ship the units 

to CE-CERT to arrive at CE-CERT earlier than September 1

5. Meeting at CE-CERT on September 22

st

nd to observe and check PEMS installation by CECERT.
307

SwRI Project 14936.12
308

SwRI Project 14936.12

PM Measurement Allowance Steering Committee LiveMeeting

August 20, September 11, 2009

1. SwRI presented the Monte Carlo Simulation (See Presented at FTP website)

2. The measurement allowance was determined based on Method 2 using Sensors’ 

PPMD

3. The measurement allowance was 0.00605 g/hp-hr

4. Sensors’ PPMD was chosen for CE-CERT in-use testing

5. It was agreed by the MASC that the AVL MSS will also participate in in-use testing 

along with the PPMD, just like it was done in the laboratory

6. The MASC requested the following from SwRI

a. plot the Sensors and Horiba 95th

b. show the results of the simulation based on reference data available within the 

concentration range obtained in the laboratory

delta on the same plot

c. plot the 95th, 50th, and 5th

d. Refreshment on the regression rules for the validation deltas

, for the validation

7. Address Item 6 above in a Livemeeting on September 11

8. The requests in Item 6 above were addressed in a livemeeting on September 11 (see 

presentation on FTP website)

a. The MASC requested that the validation deltas for the 95th, 50th, and 5th

b. The LOESS fit should be done on Sensors’ validation deltas based on Method 

1 and Method 2.

be 

regressed using the LOESS fitting rule since the criteria set for linear 

regression is not met.

c. SwRI should present the regression the CE-CERT September 22 meeting.
309

SwRI Project 14936.12

PM Measurement Allowance Steering Committee Meeting

February 17-18, 2010, US EPA, Ann Arbor

Sensors gave a presentation, proposing a correction factor for the exhaust flow. 

• Sensors calibrated the flow meter in house with the same wrong pressure configuration in 

the field 

• Sensors determined a correction factor of 1.52 for exhaust flow correction for Unit 3. 

This correction factor was accepted by the MASC

• A similar calibration will be done on Unit 2 to determine the exhaust flow correction 

factor to be used for this unit

Sensors indicated that additional corrections related to bypass flow that was not corrected for 

during the in-use testing is needed:

• The sample flow during NTE is determined as the difference between bypass flow before 

sampling and bypass flow during sampling.

• If the bypass flow is not corrected for barometric pressure during sampling, and the 

barometric changes during the long NTEs, then there would be an error introduced to the 

sample flow. 

AVL discussed the approach they took to calculate the brake-specific PM emissions using 

Method 2. The ECM broadcast fuel term in the equation was essentially frozen if it went below 

5% of the max fuel flow encountered during the test. However, to move forward with Method 2 

calculation, the MASC agreed that to the following:

• The MAX fuel flow provided by the engine manufacturer should be used not the max 

fuel determined during a test. Thus, the engine manufacturer of the CE-CERT vehicle 

should provide the information to CE-CERT to do proper Method 2 calculation

• The fuel flow along with the gas concentration terms in the equation will be frozen if the 

ECM fuel flow dipped below 10% of the MAX fuel flow.

CE-CERT gave a presentation on the different exhaust flow correction attempts they made. See 

CE-CERT presentation at the FTP website.
310

SwRI Project 14936.12

SwRI presented validation for PPMD Unit 2 and 3 using Method 1 with an exhaust flow 

correction of 1.52. The PPMD based on Method 1 failed the validation criteria. SwRI also 

showed different scenarios for the AVL, including Method 1, 2, and 3. The presentation is listed 

on the FTP website.

Below is a summary of the action items for CE-CERT and SwRI as a results of this meeting. 

These action items were compiled by Chris Laroo:

Follow-up Work to Finish PM MA Test Program

1) Agreed to exhaust flow correction factor of 1.52 for unit #3.

2) Will use exhaust flow correction factor for unit #2 that comes out of Sensors check of the 

unit #2 flow meter on their flow stand (value anticipated to be at or near 1.52 but needs to 

be determined). Value is anticipated to be known the week of March 1st

.

3) Correct PPMD data for incorrect setting in QCM bypass flow. Software setting change 

from 0 to 1.

4) Use Semtech DS barometric pressure readings to account for altitude change effects on 

data. If it is determined that the PPMD barometric pressure reading is faulty, we will 

request that Sensors upgrade the sensor to the quality used in the DS. This could be 

deemed a special source of error and accounted for via a future hardware improvement.

5) PPMD Unit #1 data will be reported for single crystal use only to reflect that fact that the 

Mass Sensitivity was incorrectly entered into the software during testing. This single 

crystal use data will then be pooled with the multi-crystal use results from units #2 and #3 

to determine the final validation % results.

6) Report PPMD Unit #2 and #3 deltas for single crystal use only as a probing exercise to 

see if multiple crystal use has an effect on mass loss. The intent of this exercise is to 

gauge the effectiveness of proposed fixes by Sensors to eliminate the PPMD low bias for 

future pilot and compliance program testing. This data will be pooled with the unit #1 
311

SwRI Project 14936.12

single crystal use results for plotting in the validation window for experimental purposes 

only. These results will not be used to determine validation.

7) Method #2, if the fuel flow rate drops below 10% of manufacturer declared maximum 

fuel rate value, then the ratio of the emission concentration terms and the ECM broadcast 

fuel rate will be frozen at that value.

8) CE-CERT will reprocess all PPMD and MSS data with correct factors.

9) CE-CERT will write a final report to be reviewed by the steering committee before 

finalization.

10) SwRI will place revised CE-CERT data in validation windows and calculate new 

validation percentages. CE-CERT should have the data to SwRI by the end of March.

11) SwRI will also place revised CE-CERT single crystal usage data in validation windows 

and calculate new validation percentages (for experimental purposes only). This again 

will help gauge whether or not there is an effect on mass loss from multiple crystal usage.

12) SwRI will write a final report to be reviewed by the steering committee before 

finalization.