PM-PEMS Measurement Allowance 

Determination

Final Report
EPA-420-R-10-902

August 2010

Assessment and Standards Division

Office of Transportation and Air Quality

U.S. Environmental Protection Agency

and

California Air Resources Board

and

Engine Manufacturers Association

Prepared by

Southwest Research Institute

SwRI Project 03.14936.12

PM-PEMS Measurement Allowance 

Determination

Final Report
PM- PEMS MEASUREMENT ALLOWANCE 

DETERMINATION

FINAL REPORT

SwRI® Project 03.14936.12

Prepared for

U.S. Environmental Protection Agency

California Air Resources Board

Engine Manufacturers Association

June 2010

Prepared by: Approved by:

 

__________________________________ ________________________________ 

Imad A. Khalek, Program Manager Jeff J. White, Director 

DEPARTMENT OF EMISSIONS RESEARCH AND DEVELOPMENT

ENGINE, EMISSIONS AND VEHICLE RESEARCH DIVISION

This report shall not be reproduced, except in full, without the written approval of Southwest Research Institute®

Results and discussion given in this report relate only to the test items described in this report.

.
REPORT 03.14936.12 ii

FOREWORD

The P M-PEMS measurement al lowance pr ogram was performed by t he Department of 

Emissions R esearch a nd D evelopment unde r Mr. Jeff W hite, Director. Dr. Imad Khalek, 

Program M anager, was t he P rincipal Investigator and Project M anager, Mr. Thomas Bougher, 

Research Engineer, was the Project Leader, and Mr. Daniel Preece, Research Assistant, was the 

laboratory te chnical a ssistant. Dr. Robert Mason, I nstitute Analyst, was t he P rincipal CoInvestigator responsible for statistical analysis, and Ms. Janet Buckingham, Staff Analyst, was 

the Project Leader/Statistics. Other SwRI Emissions R&D staff with contribution to the project 

were Mr. Michael F eist, Senior R esearch Engineer, Mr. Richard Mechler, Senior R esearch 

Technologist, Mr. Donald Parker, Senior Technician, Mr. Jose Sosa, Principal Technician, Mr. 

Keith Echtle, Laboratory A ssistant Manager, a nd M r. E rnest Krueger, L aboratory M anager. 

Additional SwRI assistance during Environmental Testing was provided by Rick Pitman, Senior 

Engineering T echnologist, Mr. Mike Negrete, S enior T echnician, Mr. David S mith, Staff 

Technician, M r. Herbert W alker, S enior E ngineering T echnologist, and M r. E ric D ornes, 

Principal Engineer. 

This w ork w as pe rformed unde r E nvironmental P rotection A gency ( EPA) Work 

Assignments 2-7, 3 -7, 4 -7, a nd 2 -12, und er SwRI P roject Numbers 12859.07, 13749.07, 

14658.07, a nd 14936.12, r espectively. T he or iginal E PA W ork Assignment Ma nager w as Mr . 

Matthew Spears. A fter March of 2009, t he E PA W ork A ssignment M anager b ecame M r. 

Christopher Laroo. This work started in J une of 2007 a nd will end b y J une 26, 2010. T esting 

started in June of 2008 and ended in September of 2009. 

 Funding for t his w ork was pr ovided by U.S E PA, Engine M anufacturers A ssociation 

(EMA), a nd the C alifornia A ir R esources B oard ( CARB). Funding b y EMA a nd C ARB w as 

provided directly to EPA in support of this work. 

In-kind engine and technical support were provided b y Volvo Powertrain. In-kind PMPEMS and technical support were provided by AVL, Horiba, and Sensors. SwRI acknowledges 

the following individuals for their laboratory technical and logistical support: 

Mr. Jeffrey Saxon and Mr. Steven Trevitz, from Volvo Powertrain 

 Mr. Scott Porter, Dr. Michael Akard and Dr. Qiang Wei, from Horiba 

Mr. William Silvis, Mr. Siegfried Roeck, Dr. Wolfgang Schindler, Dr. Roland Wanker, 

 Dr. Michael Arndt, Mr. P.J. Pankratz, and Ms. Sarah Kingham, from AVL 

 Dr. Andrew Reading, Dr. David Booker, Dr. Atul Shah, Mr. Carl Ensfield,

Mr. Timothy Bottomley, and Mr. Kevin Bouma, from Sensors 
REPORT 03.14936.12 iii

A m easurement allowance s teering committee ( SC) com posed of E MA, E PA, CARB, 

and P M-PEMS m anufacturer m embers m et on a r egular basis t hroughout t he e ntire pr oject t o 

discuss the progress made and make recommendations. The SC has contributed significantly to 

this project. SwRI acknowledges the following SC members for their active participation: 

Mr. Rey Agama, Caterpillar

Dr. Michael Akard, Horiba 

Dr. Dipak Bishnu, CARB 

Dr. David Booker, Sensors 

Dr. Bruce Cantrell, EPA Consultant 

Mr. Timothy French, EMA 

Dr. Robert Giannelli, EPA

Mr. Denny Hao, PACCAR 

Dr. Kent Johnson, CE-CERT

Mr. Craig Kazmierczak, Detroit Diesel

Mr. John Kegebein, John Deere

Mr. Thomas Kramer, Navistar

Mr. Christopher Laroo, EPA 

Mr. Hector Maldonado, CARB 

Mr. William Martin, Cummins

Dr. Shirish Shimpi, Cummins 

Mr. William Silvis, AVL

Ms. Carol Smith, Isuzu 

Mr. Matthew Spears, EPA

Mr. Steven Trevitz, Volvo Powertrain

Dr. Qiang Wei, Horiba 
REPORT 03.14936.12 iv

TABLE OF CONTENTS

Page

FOREWORD .................................................................................................................................. ii

LIST OF FIGURES .................................................................................................................... vii 

LIST OF TABLES ....................................................................................................................... xvi 

ACRONYMS ............................................................................................................................... xix

EXECUTIVE SUMMARY ......................................................................................................... xxi 

1.0 INTRODUCTION ..............................................................................................................1 

2.0 MODELING APPROACH ..................................................................................................3 

2.1 Purpose of Model .................................................................................................................3 

2.2 Model Improvement.............................................................................................................3 

2.3 Monte Carlo Simulation Approach ......................................................................................4 

2.4 Calculation Methods ............................................................................................................4 

2.4.1 Calculation Method 1 – "Exhaust Flow-Torque-Speed" Method ........................................5 

2.4.2 Calculation Method 2 – "Exhaust and Fuel Flow-Torque-Speed" Method .........................6 

2.4.3 Calculation Method 3 – "Fuel Flow-Torque-Speed" Method..............................................7 

2.5 Reference NTE Events .........................................................................................................8 

2.6 Error Surface Generation ...................................................................................................12 

2.6.1 PEMS vs. Laboratory Nominal Results .............................................................................12 

2.6.2 (PEMS-Laboratory) Deltas vs. Lab ...................................................................................14 

2.6.3 Variability Index vs. (PEMS-Laboratory) Deltas and Lab Nominal .................................15 

2.7 Error Surface Sampling and Interpolation .........................................................................18 

2.8 Brake-Specific Emissions Calculations .............................................................................20 

2.9 Convergence and Number of Trials ...................................................................................22 

2.10 Simulation Output ..............................................................................................................23 

2.11 Step-by-Step Simulation Example .....................................................................................24 

2.12 Measurement Allowance Generation .................................................................................27 

2.12.1 Regression Method.............................................................................................................27

2.12.2 Median Method ..................................................................................................................27

2.13 Model Validation ...............................................................................................................28 

3.0 PART 1065 PEMS AND LABORATORY AUDIT .........................................................32 

3.1 1065 Lab Audit ..................................................................................................................32 

3.1 1065 PEMS Audit ..............................................................................................................34 

3.2.1 Horiba Flow Audits............................................................................................................34

3.2.2 Exhaust Flow .....................................................................................................................37
REPORT 03.14936.12 v 

TABLE OF CONTENTS (CONT'D)

Page

4.0 ENGINE DYNAMOMETER TESTING AND RESULTS ..............................................40 

4.1 Testing Objective ...............................................................................................................40 

4.2 Experimental Setup ............................................................................................................40 

4.2.1 Engine and Sampling System .............................................................................................40

4.2.2 Sensors PPMD ...................................................................................................................43

4.2.3 AVL MSS ............................................................................................................................45

4.2.4 Horiba TRPM.....................................................................................................................46

4.3 Bypass Mixing Verification ...............................................................................................47 

4.4 PEMS Loss Corrections .....................................................................................................50 

4.4.1 Sensors PPMD Loss Correction ........................................................................................50

4.4.2 AVL MSS Loss Correction .................................................................................................50

4.4.3 AVL MSS Total PM Correction .........................................................................................51 

4.5 Steady-State Testing Procedure .........................................................................................51 

4.6 Data Yield During Steady-State Testing ............................................................................58 

4.6.1 Data Yield During Steady-State Testing ............................................................................59 

4.7 Accounting for CVS Variability During Steady-State Testing .........................................62 

4.8 Steady-State Testing Results ..............................................................................................62 

4.8.1 Comparison Between PEMS and Lab Delta PM ...............................................................63 

4.8.2 Correlation Between PEMS and Lab PM ..........................................................................70 

4.8.3 Steady-State PM Error Surfaces ........................................................................................73 

4.9 Transient Engine Results ...................................................................................................76 

4.10 CE-CERT Mobile Lab Correlation ....................................................................................86 

4.11 Investigation of DPF Regeneration ....................................................................................89 

4.12 Investigation of Storage and Release .................................................................................91 

4.13 Engine Manufacturers Torque and Fuel Error Surfaces ....................................................93 

5.0 ENVIRONMENTAL TESTING AND RESULTS ...........................................................96 

5.1 Reference Measurement Testing........................................................................................96 

5.2 Pressure Chamber Testing ...............................................................................................102 

5.3 Temperature and Humidity Chamber Testing .................................................................109 

5.4 Electromagnetic and Radio Frequency Interference Screening .......................................116 

5.4.1 Bulk Current Injection .....................................................................................................117

5.4.2 Radiated Immunity ...........................................................................................................122

5.4.3 Electrostatic Discharge ...................................................................................................125

5.4.4 Conducted Transients ......................................................................................................126

5.5 Vibration Testing .............................................................................................................128 

6.0 MODELNG RESULTS ...................................................................................................137 

6.1 Convergence Results from MC Runs ..............................................................................137 

6.2 Sensitivity Based on Bias and Variance ..........................................................................142 

6.3 Validation Results ............................................................................................................150 

6.4 Measurement Error Allowance Results ...........................................................................160 
REPORT 03.14936.12 vi 

TABLE OF CONTENTS (CONT'D)

Page

7.0 SUMMARY .....................................................................................................................173 

8.0 REFERENCES ................................................................................................................174 

APPENDICES Page Count

 A TEST PLAN TO DETERMINE PEMS MEASUREMENT ALLOWANCE FOR

THE PM EMISSIONS REGULATED UDNER THE MANUFACFTURER-RUN

HEAVY-DUTY DIESEL ENGINE IN-USE TESTING PROGRAM ..............................46 

 B STEERING COMMITTEE MEETING MINUTES ..........................................................33 

 C CRYSTAL BALL OUTPUT FILE DESCRIPTIONS .......................................................6 

 D MONTE CARLO SPREADSHEET COMPUTATIONS .................................................15 

 E PEMS OPERATION LOG ................................................................................................11 
REPORT 03.14936.12 vii 

LIST OF FIGURES

Figure Page

 1 Method 1 Ideal BSPM Values for Reference NTE Events ..................................................9 

 2 Error Surface Construction: PEMS vs. Laboratory Results..............................................14 

 3 Error Surface Construction: (PEMS - Lab) vs. Laboratory Results .................................15 

 4 Error Surface Construction: Error at Variability Index for 5th and 95th percentiles 

 vs. Laboratory Results .......................................................................................................16 

 5 Truncated Normal Distribution Percentiles .......................................................................17 

 6 Error Surface Construction: Error at Variability Index for 1st and 99th Percentiles 

 vs. Laboratory Results .......................................................................................................18 

 7 Truncated Standard Normal at 1st and 99th Percentiles and Uniform Probability 

 Density Functions ..............................................................................................................19 

 8 Steady-State PM Error Surface for AVL With Example Sampling for a Reference

 NTE Event .........................................................................................................................20 

 9 Overview of Monte Carlo Simulation for BSPM ..............................................................24 

 10 Error Surfaces Included in Monte Carlo Simulation .........................................................26 

 11 Linear Regression Fit To 5th and 95th Percentile Deltas ..................................................30 

 12 Loess Regression Fit To 5th and 95th Percentile Deltas ...................................................31 

 13 Linearity Check on PEMS-1 Exhaust Flow During Steady-State Engine Testing ............38 

 14 Linearity Check on PEMS-2 Exhaust Flow During Steady-State Engine Testing ............38 

 15 Linearity Check on PEMS-3 Exhaust Flow During Steady-State Engine Testing ............39 

 16 Volvo MP7 Installed in a CVS Test Cell ...........................................................................40 

 17 DPF Bypass With DOC .....................................................................................................41 

 18 Schematic of Engine Dynamometer Experimental Setup ..................................................42 

 19 PPMD Installed in the Test Cell ........................................................................................44 
REPORT 03.14936.12 viii

LIST OF FIGURES (CONT'D)

Figure Page

 20 The AVL MSS ...................................................................................................................45 

 21 The Probe and TPA for the Horiba OBS-TRPM ...............................................................46 

 22 The Horiba OBS-TRPM and OBS-2200 ...........................................................................47 

 23 Experimental Setup for Mixing Verification .....................................................................48 

 24 Mixing Verification Sample Locations ..............................................................................48 

 25 Mixing Verification Results ...............................................................................................49 

 26 Speed and Load for the 80 Points Cycle ............................................................................52 

 27 Final Six Steady-State Modes ............................................................................................53 

 28 Example of Steady-State Cycle .........................................................................................55 

 29 Steady-State Sample Times ...............................................................................................56 

 30 CVS Filter Weight Gain for Steady-State Testing .............................................................56 

 31 Steady-State Exhaust PM Concentration (μg/Mol) ...........................................................57 

 32 Steady-State Brake-Specific PM, CVS Filter (Mg/Hp-Hr) ...............................................58 

 33 Number of Valid Data Points for Steady-State Testing .....................................................59 

 34 Horiba Sample Flow to the Filter and Dilution Flow While Compressor Stops ...............61 

 35 Sensors EFM Zero During Steady-State Cycle .................................................................61 

 36 Horiba-1 Steady-State PM Concentration Deltas ..............................................................64 

 37 Sensors-1 Steady-State PM Concentration Deltas .............................................................64 

 38 AVL PM Concentration Deltas for Steady-State Testing on PEMS 1 ..............................65 

 39 Horiba-2 Steady-State PM Concentration Deltas ..............................................................65 

 40 Sensors-2 Steady-State PM Concentration Deltas .............................................................66 
REPORT 03.14936.12 ix

LIST OF FIGURES (CONT'D)

Figure Page

 41 Avl-2 Steady-State PM Concentration Deltas ...................................................................66 

 42 Horiba-3 Steady-State PM Concentration Deltas ..............................................................67 

 43 Sensors-3 Steady-State PM Concentration Deltas .............................................................67 

 44 AVL-3 Steady-State PM Concentration Deltas .................................................................68 

 45 Steady-State Concentration Deltas for Horiba ...................................................................69 

 46 Steady-State Concentration Deltas for Sensors .................................................................69 

 47 Steady-State Concentration Deltas for Avl ........................................................................70 

 48 Steady-State Horiba PEMS PM Concentration Versus the Laboratory Reference ...........71 

 49 Steady-State Sensors PEMS PM Concentration Versus the Laboratory Reference ..........71 

 50 Steady-State AVL PEMS PM Concentration Versus the Laboratory Reference ..............72 

 51 Linear Regression Correlation Between PEMS and Lab ...................................................72 

 52 Steady-State Concentration Deltas for Horiba ...................................................................73 

 53 Steady-State Concentration Deltas for Sensors .................................................................74 

 54 Steady-State Concentration Deltas for AVL......................................................................74 

 55 Final Steady-State PM Error Surface – Horiba..................................................................75 

 56 Final Steady-State PM Error Surface – Sensors ................................................................75 

 57 Final Steady-State PM Error Surface – AVL ....................................................................76 

 58 Repeat Engine Speed Traces for NTE Transient Cycle .....................................................77 

 59 Repeat Engine Torque Traces for NTE Transient Cycle ...................................................78 

 60 Repeat AVL Soot Concentration Traces for NTE Transient Cycle ...................................79 

 61 AVL Soot Concentration During NTE Transient Cycles, Events 20-23 ...........................79 
REPORT 03.14936.12 x 

LIST OF FIGURES (CONT'D)

Figure Page

 62 Comparison of AVL and Horiba Real Time Signals During Transient Cycle ..................80 

 63 CVS BSPM and Correction Factor for Transient Cycle ....................................................81 

 64 Horiba Concentration Deltas for Transient Engine Testing ..............................................82 

 65 Sensors Concentration Deltas for Transient Engine Testing .............................................82 

 66 AVL Concentration Deltas for Transient Engine Testing .................................................83 

 67 Final Horiba Transient PM Error Surface ..........................................................................84 

 68 Final Sensors Transient PM Error Surface ........................................................................85 

 69 Final AVL Transient PM Error Surface .............................................................................85 

 70 Exhaust Configuration for CE-CERT Correlation .............................................................86 

 71 Brake-Specific PM Results From CE-CERT Correlation .................................................88 

 72 CVS Filter Weight Gain During Tunnel Blanks ................................................................88 

 73 Brake Specific PM Emissions During Active Regeneration .............................................90 

 74 Total Exhaust Number Concentration During Storage and Release ..................................92 

 75 Brake-Specific PM Emissions During Storage and Release Investigation ........................92 

 76 OEM Supplied Torque Errors ............................................................................................94 

 77 OEM Supplied Fuel Flow Errors .......................................................................................94 

 78 Target Dilution Ratio and PM Level Profile for Environmental Testing ..........................97 

 79 Experimental Setup for Environmental Testing ................................................................98 

 80 Horiba Environmental Baseline Measurements.................................................................99 

 81 Sensors Environmental Baseline Measurements ...............................................................99 

 82 AVL Environmental Baseline Measurements .................................................................100 
REPORT 03.14936.12 xi

LIST OF FIGURES (CONT'D)

Figure Page

 83 Reference MSS Environmental Baseline Measurements ................................................100 

 84 Reference AVL Versus PEMS AVL for Environmental Baseline ..................................101 

 85 Original and Revised Profile for Altitude Testing ...........................................................102 

 86 Altitude Testing Chamber ................................................................................................103 

 87 PEMS Installed in the Altitude Chamber.........................................................................104 

 88 Horiba Environmental Pressure Measurements ...............................................................104 

 89 Horiba Total Flow and Dilution Ratio During Pressure Testing .....................................105 

 90 Sensors Environmental Pressure Measurements .............................................................106 

 91 AVL Environmental Pressure Measurements ..................................................................106 

 92 AVL Pressure Median Versus MAD ...............................................................................107 

 93 Horiba Pressure Median Versus MAD ............................................................................107 

 94 Sensors Pressure Median Versus MAD ...........................................................................108 

 95 Final Error Surface for Environmental Pressure AVL PM Concentration ......................109 

 96 Temperature and Humidity Profile for Environmental Testing .......................................110 

 97 PEMS Installed in the Temperature and Humidity Chamber ..........................................110 

 98 The PM Generator Installed Outside the Temperature and Humidity Chamber .............111 

 99 Horiba Environmental Temperature Measurements ........................................................112 

 100 Horiba Temperature during Environmental Temperature Testing ..................................112 

 101 Sensors Environmental Temperature Measurements.......................................................113 

 102 AVL Environmental Temperature Measurements ...........................................................113 

 103 Horiba Temperature Median Versus MAD .....................................................................114 
REPORT 03.14936.12 xii

LIST OF FIGURES (CONT'D)

Figure Page

 104 Sensors Temperature Median Versus MAD ....................................................................115 

 105 Horiba Temperature and Humidity Median Versus MAD ..............................................115 

 106 Final Error Surface Environmental Temperature and Humidity AVL PM .....................116 

 107 AVL PEMS in the Radiation Chamber for EMI and RFI Testing ...................................117 

 108 Bulk Current Injection Probe ...........................................................................................118 

 109 Horiba PEMS Setup for Bulk Current Injection ..............................................................118 

 110 Sensors Setup During Bulk Current Injection Testing ....................................................119 

 111 AVL Setup During Bulk Current Injection Testing .........................................................119 

 112 Horiba Flow Disturbance From Bulk Current Injection ..................................................120 

 113 Horiba Exhaust Flow Noise on Analog Cable During Bulk Current Injection ...............121 

 114 BCI Noise on AVL Analog Output Cable .......................................................................122 

 115 Horiba PEMS Setup During Radiated Immunity Testing................................................123 

 116 Sensors PEMS Setup During Radiated Immunity Testing ..............................................124 

 117 AVL PEMS Setup During Radiated Immunity Testing ..................................................124 

 118 Horiba Dilution Ratio Fluctuations During Radiated Immunity .....................................125 

 119 Electrostatic Discharge Simulator....................................................................................126 

 120 Sensors Setup During Conducted Transients Testing ......................................................127 

 121 AVL Setup During Conducted Transients Testing ..........................................................127 

 122 Sensors PEMS Mounted for Transverse Horizontal Vibration........................................128 

 123 Sensors PEMS Mounted for Longitudinal Horizontal Vibration ....................................129 

 124 Sensors PEMS Mounted for Transverse 45° Vibration ...................................................129 
REPORT 03.14936.12 xiii

LIST OF FIGURES (CONT'D)

Figure Page

 125 Sensors PEMS Mounted for Longitudinal 45° Vibration ................................................130 

 126 Sensors PEMS Mounted for Vertical Vibration ..............................................................130 

 127 Sensors PEMS Mounted for 45° Vertical Vibration ........................................................131 

 128 Power Spectral Density for Vertical Vibration Testing ...................................................132 

 129 Power Spectral Density for Horizontal Vibration Testing ...............................................132 

 130 Horiba PEMS Vibration Positions ...................................................................................133 

 131 Sensors Total Flow During Vibration Testing .................................................................134 

 132 AVL PEMS Vibration Positions ......................................................................................135 

 133 AVL Soot Measurement Noise During Vibration Testing ..............................................135 

 134 Convergence for AVL Method 1 As a Percent of BSPM Threshold ...............................138 

 135 Convergence for AVL Method 2 As a Percent of BSPM Threshold ...............................138 

 136 Convergence for AVL Method 3 As a Percent of BSPM Threshold ...............................139 

 137 Convergence for Horiba Method 1 As a Percent of BSPM Threshold ............................139 

 138 Convergence for Horiba Method 2 As a Percent of BSPM Threshold ............................140 

 139 Convergence for Sensors Method 1 As a Percent of BSPM Threshold ..........................141 

 140 Convergence for Sensors Method 2 As a Percent of BSPM Threshold ..........................141 

 141 Box Plot of Error Surface Sensitivity Based on Bias and Variance for AVL

 BSPM Method 1 ..............................................................................................................146 

 142 Box Plot of Error Surface Sensitivity Based on Bias and Variance for

 AVL Method 2 .................................................................................................................146 

 143 Box Plot of Error Surface Sensitivity Based on Bias and Variance for

 AVL Method 3 .................................................................................................................147 
REPORT 03.14936.12 xiv

LIST OF FIGURES (CONT'D)

Figure Page

 144 Box Plot of Error Surface Sensitivity Based on Bias and Variance for Horiba

 BSPM Method 1 ..............................................................................................................147 

 145 Box Plot of Error Surface Sensitivity Based on Bias and Variance for Horiba

 BSPM Method 2 ..............................................................................................................148 

 146 Box Plot of Error Surface Sensitivity Based on Bias and Variance for Sensors 

 BSPM Method 1 ..............................................................................................................148 

 147 Box Plot of Error Surface Sensitivity Based on Bias and Variance for Sensors 

 BSPM Method 2 ..............................................................................................................149 

 148 Validation Percentiles for the 141 Reference NTE Events for AVL Method 1 ..............151 

 149 Validation Percentiles for the 141 Reference NTE Events for AVL Method 2 ..............152 

 150 Validation Percentiles for 141 Reference NTE Events for AVL Method 3 ....................152 

 151 Validation Percentiles for 141 Reference NTE Events for Horiba Method 1 .................153 

 152 Validation Percentiles for 141 Reference NTE Events for Horiba Method 2 .................153 

 153 Validation Percentiles for 141 Reference NTE Events for Sensors Method 1 ................154 

 154 Validation Percentiles for 141 Reference NTE Events for Sensors Method 2 ................154 

 155 Validation 95th Percentile BSPM Deltas Loess Fit for Sensors Method 1 .....................155 

 156 Validation 5th Percentile BSPM Deltas Loess Fit for Sensors Method 1 .......................156 

 157 Validation 95th Percentile BSPM Deltas Loess Fit for Sensors Method 2 .....................156 

 158 Validation 5th Percentile BSPM Deltas Loess Fit for Sensors Method 2 .......................157 

 159 Validation 95th Percentile BSPM Deltas Loess Fit for AVL Method 1 ..........................157 

 160 Validation 5th Percentile BSPM Deltas Loess Fit for AVL Method 1............................158 

 161 Validation 95th Percentile BSPM Deltas Loess Fit for AVL Method 2 ..........................158 

 162 Validation 5th Percentile BSPM Deltas Loess Fit for AVL Method 2............................159 
REPORT 03.14936.12 xv

LIST OF FIGURES

Figure Page

 163 Validation 95th Percentile BSPM Deltas Loess Fit for AVL Method 3 ..........................159 

 164 Validation 5th Percentile BSPM Deltas Loess Fit for AVL Method 3............................160 

 165 Regression Plot of 95th Percentile Delta BSPM Versus Ideal BSPM for

 AVL Method 1 .................................................................................................................162 

 166 Regression Plot of 95th Percentile Delta BSPM Versus Ideal BSPM for AVL

 Method 2 ..........................................................................................................................162 

 167 Regression Plot of 95th Percentile Delta BSPM Versus Ideal BSPM for AVL

 Method 3 ..........................................................................................................................163 

 168 Regression Plot of 95th Percentile Delta BSPM Versus Ideal BSPM for Horiba 

 Method 1 ..........................................................................................................................164 

 169 Regression Plot of 95th Percentile Delta BSPM Versus Ideal BSPM for Horiba 

 Method 2 ..........................................................................................................................165 

 170 Regression Plot of 95th Percentile Delta BSPM Versus Ideal BSPM for Sensors

 Method 1 ..........................................................................................................................166 

 171 Regression Plot of 95th Percentile Delta BSPM Versus Ideal BSPM for Sensors

 Method 2 ..........................................................................................................................167 

 172 Validation On-Road and Regression Functions Based on the Simulation Model for 

 BSPM Sensors Method 1 With No Regen .......................................................................170 

 173 Validation On-Road and Regression Functions Based on the Simulation Model for 

 BSPM Sensors Method 2 With No Regen .......................................................................170 

 174 Validation On-Road and Regression Functions Based on the Simulation Model for 

 BSPM AVL Method 1 With No Regen ...........................................................................171 

 175 Validation On-Road and Regression Functions Based on the Simulation Model for 

 BSPM AVL Method 2 With No Regen ...........................................................................171 

 176 Validation On-Road and Regression Functions Based on the Simulation Model for 

 BSPM AVL Method 3 With No Regen .......................................................................... 172 
REPORT 03.14936.12 xvi 

LIST OF TABLES

Table Page

 1 PM-PEMS Used along with Serial Number ........................................................................2 

 2 Reference NTE Events and Ideal BSPM Emissions ..........................................................10 

 3 Descriptive Statistics for BSPM Emissions for 141 Reference NTE Events ....................10 

 4 Input Parameters For Reference NTE Events ....................................................................11 

 5 Error Surfaces for Monte Carlo Simulation .......................................................................13 

 6 Error Surfaces Used for Computing Brake-Specific PM Emissions by Three 

 Calculation Methods ..........................................................................................................22 

 7 Example of Selection of Measurement Allowance at 0.02 g/hp-hr NTE Threshold for 

 the AVL PEMS ..................................................................................................................28 

 8 Linearity Verification Results for Intake Air Flwo and Fuel Flow ...................................32 

 9 CVS Propane Recovery Check Summary ..........................................................................33 

 10 Linearity Verification for PM Balance ..............................................................................33 

 11 Summary of Part 1065 Audits............................................................................................34 

 12 Linearity Verifications for Horiba PEMS ..........................................................................35 

 13 Linearity Verifications for Sensors PEMS ........................................................................36 

 14 Linearity Verific Ations for AVL PEMS ...........................................................................37 

 15 List of DPF Bypass Configurations ...................................................................................42 

 16 Sample Order for Steady-State Cycle Testing ...................................................................54 

 17 Data Yield by Each PM-PEMS..........................................................................................60 

 18 Test Procedure for CE-CERT Correlation .........................................................................87 
REPORT 03.14936.12 xvii 

LIST OF TABLES (CONT'D)

Table Page

 19 PM Emissions Results from Active DPF Regeneration ....................................................89 

 20 Average Brake-Specific Emissions During Storage and Release Cycle ...........................93 

 21 OEM Error Surface Deltas for Torque and Fuel Flow ......................................................95 

 22 Summary of Number of Reference NTEs Meeting 2% Convergence .............................141 

 

 23 Error Surface Sensitivity to Bias and Variance for 141 Reference NTE Events for

 AVL BSPM Method 1 .....................................................................................................143 

 24 Error Surface Sensitivity to Bias and Variance for 141 Reference NTE Events for 

 AVL BSPM Method 2 .....................................................................................................143 

 25 Error Surface Sensitivity to Bias and Variance for 141 Reference NTE Events for 

 AVL BSPM Method 3 .....................................................................................................143 

 26 Error Surface Sensitivity to Bias and Variance for 141 Reference NTE Events for 

 HORIBA BSPM Method 1 ..............................................................................................144 

 27 Error Surface Sensitivity to Bias and Variance for 141 Reference NTE Events for 

 HORIBA BSPM Method 2 ..............................................................................................144 

 28 Error Surface Sensitivity to Bias and Variance for 141 Reference NTE Events for 

 SENSORS BSPM Method 1 ............................................................................................145 

 29 Error Surface Sensitivity to Bias and Variance for 141 Reference NTE Events for 

 SENSORS BSPM Method 2 ............................................................................................145 

 30 Summary of Error Surface Sensitivities to Bias and Variance for BSPM Method 1 ......149 

 31 Summary of Error Surface Sensitive to Bias and Variance for BSPM Method 2 ..........150 

 32 Summary of Error Surface Sensitive to Bias and Variance for BSPM Method 3 ...........150 

 33 Loess Smoothing Parameters for Validation Percentiles .................................................155 
REPORT 03.14936.12 xviii

LIST OF TABLES (CONT'D)

Table Page

 34 Measurement Error at Threshold for BSPM Using Regression and Median Methods 

 for AVL Method 1 ...........................................................................................................161 

 35 Measurement Error at Threshold for BSPM Using Regression and Median Methods 

 for AVL Method 2 ...........................................................................................................162 

 36 Measurement Error at Threshold for BSPM Using Regression and Median Methods 

 for AVL Method 3 ...........................................................................................................163 

 37 Measurement Error at Threshold for BSPM Using Regression and Median Methods for 

 HORIBA Method 1 ..........................................................................................................164 

 38 Measurement Error at Threshold for BSPM Using Regression and Median Methods 

 for HORIBA Method 2 ....................................................................................................165 

 39 Measurement Error at Threshold for BSPM Using Regression and Median Methods for 

 SENSORS Method 1........................................................................................................166 

 40 Measurement Error at Threshold for BSPM Using Regression and Median Methods for

 SENSORS Method 2........................................................................................................167 

 41 BSPM Measurement Error in Percent of NTE Threshold by PEMS and Calculation 

 Method .............................................................................................................................168 

 42 Measurement Allowance at NTE Threshold by Emissions for Method 2 .......................169 

 43 Summary of BSPM Model Validation Results ................................................................172 
REPORT 03.14936.12 xix

ACRONYMS

ACES Advanced Collaborative Emissions Study 

BCI Bulk Current Injection 

BS Brake-Specific

BSFC Brake-Specific Fuel Consumption 

BSPM Brake-specific Particulate Matter

CARB California Air Resources Board

CE-CERT Bourns College of Engineering Center for Environmental Research & Technology 

CF Correction Factor 

CFR Code of Federal Regulations

COV Coefficient of Variation

CVS Constant Volume Sampling 

DCS Diffusion Charge Sensor 

DOC Diesel Oxidation Catalyst

DPF Diesel Particulate Filter

DR Dilution Ratio

EAD Electrical Aerosol Detector (TSI, Inc.)

EATS Exhaust after-treatment system

ECM Engine Control Module 

EE Electrical Enclosure

EEPS Engine Exhaust Particle Sizer (TSI, Inc.)

EFM Electronic Flow Meter

EGR Exhaust Gas Recirculation

EMA Engine Manufacturers Association 

EMI Electromagnetic Interference

EPA Environmental Protection Agency 

ESD Electrostatic Discharge

HDIUT Heavy-Duty In-Use Testing

HE Heated Enclosure

HEPA High Efficiency Particulate Air

LFE Laminar Flow Element

MA Measurement Allowance

MAD Median Absolute Deviation 

ME Mechanical Enclosure

MEL Mobile Emissions Laboratory 

MSS Micro Soot Sensor (AVL) 

NMHC Non-Methane Hydrocarbons 

NOX The Oxides of Nitrogen (NO + NO2

NTE Not-to-exceed

)\ 

OBS On-board systems 

OC/EC Organic Carbon/Elemental Carbon

PEMS Portable Emission Measurement System

PM Particulate Matter

PPMD Portable Particulate Measurement Device (Sensors, Inc.)
REPORT 03.14936.12 xx 

ACRONYMS (CONT'D)

PSD Power Spectral Density

QCM Quartz Crystal Microbalance

RFI Radio Frequency Interference 

RMS Root Mean Square 

SAE Society of Automotive Engineers 

SS Steady State 

SwRI Southwest Research Institute 

TPA Tail Pipe Adaptor 

TRPM Transient Response Particulate Matter

ULSD Ultra-Low Sulfur Diesel 

VGT Variable Geometry Turbocharger
REPORT 03.14936.12 xxi 

EXECUTIVE SUMMARY

The U nited S tates E nvironmental P rotection A gency ( EPA), E ngine M anufacturers 

Association ( EMA) a nd C alifornia A ir R esources B oard ( CARB) agreed t o pur sue an 

experimental da ta dr iven program to establish a me asurement a llowance ( MA) f or in -use 

particulate matter (PM) testing using PM portable emissions measurement systems (PM-PEMS). 

The M A i s a br ake-specific P M e missions e rror a ssociated with using in -use P M-PEMS 

equipment compared to the laboratory reference filter method. If the MA error is a positive value 

above zero, after the EPA rounding method, as described in the Code of Federal Title 40, Part 

1065 [1], it will increase the EPA in-use not-to-exceed (NTE) standard by the rounded value. If 

the error is negative or zero, it will not contribute to any changes to the in-use standard. 

The measurement allowance steering committee (SC) accepted the following PM-PEMS 

to be part of the MA program: 

• Sensors Portable Particulate Measuring Device (PPMD). This is a PM-PEMS that uses 

proportional dilution and a series of 8 quartz crystal microbalances to measure total (solid 

plus volatile) PM. The total PM is measured as a single flow-weighted value for an NTE 

event.

• Horiba Transient Particulate Matter (TRPM). This is a PM-PEMS that uses proportional 

dilution, a real time electrical aerosol detector, and an integrated filter sample to report 

total PM. The total PM is measured as a single flow-weighted value for a NTE event. 

This instrument can report real time total PM, but was not used as such on this program. 

• AVL mic ro-soot s ensor (MSS). This i s an i nstrument t hat us es c onstant di lution a nd a 

photo acoustic detector to measure soot or the elemental carbon portion of PM. Soot is 

measured in real time during an NTE event. 

The SC agreed that only the PPMD by Sensors and TRPM by Horiba would be used for 

the official determination and validation of the measurement allowance generated because both 

are designated as complete PEMS and both measure total (solid plus volatile) PM, as required by 

US EPA to be valid PM-PEMS. The SC also agreed that only the PM-PEMS that produces the 

lowest positive 95th percentile measurement allowance, based on Sensors or Horiba only, would 

be chosen for in-use validation by CE-CERT due to funding limitation. The third instrument, the 

MSS b y A VL, was not a com plete P M-PEMS a nd w as a lways us ed i n c onjunction w ith t he 

PPMD on t his pr ogram, ba sed on a n a greement r eached be tween S ensors a nd A VL. T he S C 

agreed that the MSS would not be considered as an option for official measurement allowance 

determination, unless both the PPMD and the TRPM failed validation. 

The PM-PEMS-MA project included four main elements: 

• Laboratory steady-state (SS) and transient engine NTE testing using PM-PEMS and CVS 

filter m easurement dur ing S S t esting onl y. T he S S testing was us ed t o c apture bi as, 

compared to the CVS, and the transient was used to capture precision only since there is 

no reference method to measure PM during a short NTE event. A 2007 heavy-duty diesel 

Mack M P7, from Volvo P owertrain, w as us ed t o c onduct t he e ngine e xperiments. T he 
REPORT 03.14936.12 xxii 

engine w as c onfigured w ith a b ypass a round t he di esel pa rticle f ilter ( DPF) t o pr ovide 

PM concentration levels similar to those expected at the NTE threshold limits between 

0.02 g/hp-hr and 0.03 g/hp-hr. 

• Environmental t esting s uch a s t he e ffect of s hock a nd vi bration, pr essure, t emperature 

and relative humidity, and electric noise on PM-PEMS precision. 

• Monte C arlo simulation t o de termine a br ake-specific m easurement al lowance va lue 

using error surfaces generated from the PM-PEMS laboratory and environmental testing, 

and from error surfaces generated during the gaseous PEMS program 

 ○ A total of 141 Reference NTEs were provided by EMA as an input to the model 

 for calculating ideal brake specific emissions, prior to the Monte Carlo simulation 

• Model validation using data generated from in-use PM-PEMS testing by CE-CERT.

 Three methods are used to determine in-use NTE brake-specific PM emissions:

• Method 1 f(PM�����, torquei, speedi, exhaust-flowi

• Method 2 f(PM�����, torque

) 

i, speedi, exhaust- flowi, G-flowi, fuelECMi

• Method 3 f(PM

) 

i, torquei, speedi, and fuelECMi, G-flowi ) 

Where PM����� is a f low-weighted PM m easurement, i is i nstantaneous, E CM i s engine 

control module, and G-flow is gas-based fuel flow. All methods require ECM broadcasted torque 

and speed. In addition, Method 1 r equires measured exhaust flow but not fuel flow; Method 2 

requires measured exhaust flow, ECM broadcasted fuel flow, and G-flow; Method 3 is similar to 

Method 2, but it does not require measured exhaust flow. Besides real time PM measurement, all 

methods can use a single flow-weighted PM measurement for an NTE event, except Method 3, 

where real time PM measurement is required. Thus, Method 1 and Method 2 were applied to all 

three PM-PEMS, but Method 3 was only applied to the AVL MSS. 

Compared to the Horiba TRPM, the Sensors PPMD, as shown in Table ES-1, produced 

the low est pos itive 95t h pe rcentile measurement a llowance of 0.006 05 g/hp-hr for a n NTE 

threshold l evel of 0.02 g/hp-hr, us ing c alculation M ethod 2. T he H oriba T RPM pr oduced a 

measurement allowance value of 0.0100 g/hp-hr. Thus, the PPMD was selected by the SC for inuse te sting b y C E-CERT for M onte C arlo m odel va lidation. A lthough not a ccepted a s a P MPEMS, the AVL MSS produced the lowest measurement allowance of essentially zero. The SC 

agreed to include the AVL MSS during in-use validation because it was used in conjunction with 

the PPMD during the laboratory portion of the testing. 
REPORT 03.14936.12 xxiii

TABLE ES-1. MEASRUEMENT ALLOWANCE BASED ON 0.02 G/HP-HR 

THRESHOLD

Measurement Errors at NTE Threshold, g/hp-hr

PEMS Method 1 Method 2 Method 3

AVL 0.0001 -0.0005 -0.0005 

Horiba 0.0109 0.0100 n/a 

Sensors 0.0069 0.0061 n/a 

Figures ES-1 a nd E S-2 show e xamples of va lidation pl ots f or S ensors P PMD us ing 

Method 1 and AVL MSS using Method 3. The dots on each of the two figures represent the inuse delta BSPM between PEMS and CE-CERT 47 mm filter measurements on the y-axis versus 

CE-CERT BSPM on the x-axis (Ideal BSPM determined by the filter measurement). Similarly, 

the lines represents the 5th and 95th percentile errors produced by Monte-Carlo simulation based 

on laboratory testing versus a r eference NTE BSPM value on the x-axis (Ideal BSPM using 47 

mm measurements). To pass m odel va lidation, ≤ 10% of t he i n-use N TE delta br ake-specific 

PM (BSPM) between the PEMS and the CE-CERT (dots on F igures ES1 and ES2) must reside 

outside the 95th percentile and 5th percentile lines. 

The Sensors PPMD failed validation using Method 1 a nd Method 2, a s shown in Table 

ES-2. For Method 1, 32 percent of the data were below the 5th percentile, and for Method 2, 34 

percent of the data were below the 5th percentile. The failure was mainly due to negative bias. 

The e xact c ause f or ne gative bi as i s unknow n, and f unding l imitation di d not pe rmit f urther 

investigations to resolve this issue under the scope of the MA program. 

The A VL M SS pa ssed va lidation us ing M ethod 3, a s s hown i n T able ES-3, w here

9.89 percent of the data were outside the 5th and 95th percentiles, with the majority of the failed 

points being above the 95th percentile. Method 2 failed validation by one percentage point, and 

Method 1 failed validation by having 18 percent of the data outside the 5th and 95th percentiles, 

with the majority of failed points being above the 95th percentile. 
REPORT 03.14936.12 xxiv 

FIGURE ES-1. IN-USE VALIDATION PLOT FOR SENSORS PPMD USING METHOD

1. (Y AXIS IS THE DIFFERENCE BETWEEN PEMS AND IDEAL BSPM. IDEAL BSPM 

IS THE LABORATORY CVS BSPM FOR THE 5TH AND 95TH AND THE CE-CERT 

BSPM FOR THE DOTS)

FIGURE ES-2. IN-USE VALIDATION PLOT FOR AVL MSS USING METHOD 3. (Y 

AXIS IS THE DIFFERENCE BETWEEN PEMS AND IDEAL BSPM. IDEAL BSPM IS 

THE LABORATORY CVS BSPM FOR THE 5TH AND 95TH AND THE CE-CERT 

BSPM FOR THE DOTS)
REPORT 03.14936.12 xxv 

TABLE ES-2. SENSORS PPMD VALIDATION RESULTS

Method 1 Method 2

Total No. CE-CERT Points 217 217 

No. CE-CERT Points within Ideal BSPM Range 210 210 

No. CE-CERT Points above 95th Percentile 0 1 

No. CE-CERT Points below 5th Percentile 68 70 

No. CE-CERT Points between 5th and 95th Percentiles 142 139 

% CE-CERT points that did not validate 32.38 33.81 

TABLE ES-3. AVL MSS VALIDATION RESULTS

Method 1 Method 2 Method 3

Total No. CE-CERT Points 271 271 271 

No. CE-CERT Points within Ideal BSPM Range 263 263 263 

No. CE-CERT Points above 95th Percentile 47 28 23 

No. CE-CERT Points below 5th Percentile 2 2 3 

No. CE-CERT P oints be tween 5t h a nd 9 5th 

Percentiles 222 233 237 

% CE-CERT points that did not validate 18.08 11.41 9.89 

Due t o t he l ack of additional f unding, t he measurement allowance pr ogram w as

concluded by the SC without being able to solve the lack of validations issue with the Sensors 

PPMD, or pe rform a dditional C E-CERT te sting w ith the H oriba T RPM to determine if i t 

validates the model. The AVL MSS passed the validation criteria using Method 3. H owever, 

the MSS as used in this program is not accepted as a PM-PEMS by EPA, and the measurement 

allowance generated b ased on t he pe rformance o f this instrument is not an official p art of the 

measurement allowance. 

The final official measurement allowance accepted by the SC was based on t he Sensors 

PPMD. Further investigation of why the PPMD did not validate the model is being investigated 

outside the MA program, and the details are expected to be part of the CE-CERT Final Report on 

PM-PEMS In-Use Validation Testing. 
REPORT 03.14936.12 1 of 174 

1.0 INTRODUCTION

The U.S. EPA, EMA, and CARB agreed to pursue an experimental data driven program 

to determine PEMS bias and precision measurement errors expected in in-use NTE testing and 

compliance b efore en forcement. The i dea i s t o combine t hese e rrors i nto a m easurement 

allowance which will be used with the EPA in-use regulatory standard. The combination, in-use 

standard plus m easurement al lowance, will al low f or a l arger t hreshold due t o i nstrument a nd 

measurement uncertainties that the engine manufacturers must comply with. The gaseous PEMS 

measurement a llowance pr ogram w as c ompleted b y S wRI i n A pril of 2007, a nd t his w ork 

focuses on the PM-PEMS measurement allowance, as set forth in Test Plan and meeting minutes, 

shown in Appendices A and B, respectively. 

To determine all bias and precision errors associated with in-use PM-PEMS measurement 

would take a very extensive set of experimental data and engines to cover all engine steady-state 

and transient NTE operations, as well as, different environmental conditions and configurations 

that may influence the measurement. Instead of focusing on a wide matrix of experimental data,

the SC approach was to: 

a) Perform repeats on a series of six laboratory NTE steady-state laboratory tests using the 

CVS PM filter method and the PM-PEMS, and establish a bias error surface for each of 

the PEMS tested. 

b) Perform repeats on a series of laboratory N TE t ransient c ycles c ontaining thirty 32 

seconds N TE e vents, and use t he P M-PEMS m easurement t o pr oduce p recision e rror 

surface for each of the PEMS tested. 

c) Perform a series of e nvironmental te sts tha t inc ludes e lectromagnetic a nd radio 

frequency i nterferences, shock a nd vi bration, pr essure, a nd t emperature a nd hum idity, 

and produce an error surface, if any, for each PEMS associated with each parameter. 

d) Use torque and fuel flow error surfaces provided by the engine manufacturers. 

e) Use ot her error s urfaces es tablished during t he gaseous m easurement allowance 

program. 

f) Use M onte-Carlo s imulation ba sed on a s et of 141 Ideal br ake-specific P M ( BSPM) 

reference NTE events to predict the error distribution at each reference NTE. 

g) Use t he 95 th percentile a nd 5th

h) Perform act ual i n-use testing w ith the PM-PEMS using t he C E-CERT tr ailer to 

determine whether or not the data generated in-use validates the model.

percentile of t he error di stribution a t each Ideal B SPM 

reference N TE l evel as the uppe r a nd l ow bounda ry o f t he de ltas be tween P M-PEMS 

BSPM and Ideal BSPM.. 

 A total of nine PM-PEMS, three from each manufacturer, were used on this program. The 

PM-PEMS, shown in Table 1, included the Sensors PPMD, Horiba TRPM, and AVL MSS. Each 

set of t hree P M-PEMS, one PM-PEMS f rom each manufacturer, was t ested simultaneously i n 

parallel us ing a s eries of s teady-state and transient N TE engine experiments. After t he 

completion of steady-state testing and transient testing, one PM-PEMS from each manufacturer 

was us ed on a s eries of e nvironmental t est c onditions t hat i ncluded electromagnetic and radio 

frequency i nterferences, shock a nd vi bration, atmospheric pr essure, a nd t emperature a nd 

humidity. 
REPORT 03.14936.12 2 of 174 

TABLE 1. PM-PEMS USED ALONG WITH SERIAL NUMBER

PEMS Name PEMS Serial Number

Horiba 1 10107-01 

Horiba 2 10107-02

Horiba 3 10107-03

Sensors 1 E08-PD03

Sensors 2 G08-PD02

Sensors 3 A08-PD03

AVL 1 346

AVL 2 472

AVL 3 273 

To validate the model, a series of in-use tests with the PM-PEMS was performed by CECERT us ing t he C E-CERT e missions tr ailer. Details a bout thi s te sting will be pr ovided in a 

separate report by CE-CERT on PM-PEMS In-Use Validation Testing.

This report describes: 

• Model approach used to perform the modeling portion of this work. 

• Engine ex periments with the P M-PEMS including steady-state and transient N TE P M 

results

• Environmental setup and PM results

• Measurement allowance produced by the model for each of the three PM-PEMS provided 

by each manufacturer

• Model validation using the PM-PEMS selected for in-use testing 
REPORT 03.14936.12 3 of 174 

2.0 MODELING APPROACH

2.1 Purpose of Model 

This program w as designed t o generate BSPM m easurement al lowances b ased on 

rigorous statistical methods applied to a large body of data. At the same time, it was desirable to 

exclude out lier da ta ca used by ex treme m easurement er rors w hich were not cons idered 

representative of nor mal i n-use ope rations. A di rect a pproach c ould ha ve be en t o t est P EMS 

against s ome ki nd of mobile l aboratory r eference ( such a s t he C E-CERT M obile E mission 

Laboratory) on a l arge number o f ve hicles, a nd qua ntify e rrors di rectly. H owever, s uch a n 

approach would have been expensive in terms of both time and funding. 

Given these f actors, the S teering C ommittee ultimately e lected to us e a s imulation 

approach in order to generate the BSPM measurement allowances, similar to what was done in 

the g aseous i n-use e missions t esting pr ogram [2]. I n t his a pproach, t he S teering C ommittee 

defined all of the expected sources of PEMS measurement errors based on existing in-use testing 

expertise and understanding of how the PEMS functioned. Priority w as given to the Horiba's 

and Sensors' PEMS in the design of experiments. Each of these errors was quantified using a 

series of cont rolled laboratory experiments, each de signed to isolate er rors r elated to a s ingle 

error source. The results of each experiment would essentially be an empirical model of a given 

source of measurement error. In this report, these error models are referred to as error surfaces. 

It is important to note that each of these error surfaces represents an incremental error of PEMS 

measurement, as compared to an associated laboratory reference measurement.

2.2 Model Improvement

Several i mprovements t o t he e xecution of t he M onte C arlo s imulation m odel w ere 

implemented to improve the efficiency and post-processing of the simulation runs. Eight macros 

were written to perform various tasks and are summarized in the section below. 

Macro 1: Controls batch processing and allows the ability to run several simulations backto-back (batch mode). Reads each reference NTE event data, number of trials, and the number 

of reference NTE events in the batch run. Calls other macros.

Macro 2: Clears and deletes ex tra r ows i n error m odel ( see A ppendix C f or a d etailed 

description). Calculates ideal PM emissions for each calculation method. 

Macro 3: Checks e rror s urfaces t urned ‘ off’ i n s imulation r un a nd c lears unus ed e rror 

surface cha rts and Excel t abs for c alculation s peedup (see A ppendix D ) f or a de tailed 

description). 

Macro 4: Controls Crystal Ball run preferences, runs Crystal Ball simulation for the given 

NTE Event, and controls Crystal Ball creation of Report and Extract data files. No longer stores 

each trial i c value (40,000 – 65,000 values). Reduced the number of sensitivity charts created. 

Only stores BS emissions in g/hp-hr units. T his reduced EXTRACT and REPORT files from 

133 MB to 17 MB per NTE event (≈ 87% reduction). 
REPORT 03.14936.12 4 of 174 

Macro 5: Reads EXTRACT file, controls calculation of 5th, 50th and 95th percentiles from 

full model and calls Macro 6. 

Macro 6: Computes 5th, 50th and 95th percentile delta BS emissions and confidence limits 

from order statistics from the Full model. These values are used in the measurement allowance 

plots and to check for convergence. 

Macro 7: Reads Extract file Validation data, computes 5th, 50th and 95th percentiles for the 

Validation model (calls Macro 6). 

Macro 8: Reads RE PORT file, selects a nd formats s ensitivity da ta f or A ssumption 

sensitivities greater than 5 percent or less than -5 percent contribution to variance and stores in 

sensitivity file. Creates sensitivity file with “important” error surfaces. 

The de velopment of t he m acros r esulted i n s ubstantial i mprovements i n t he s imulation 

run-time and post-processing of the extract files. The estimated reduction in model simulation 

run-time and post processing was approximately 80 to 85 percent of the time used in the gaseous 

PEMS program. The developed macros eliminated the need to manually post-process the files as 

had been done in the gaseous PEMS program. The batch processing allowed up to 20 reference 

NTE events to be simulated in a single Excel run. The size of the EXTRACT and REPORT files 

was greatly reduced by only storing needed variables and delta PM emissions.

2.3 Monte Carlo Simulation Approach 

The e rror s urfaces r epresenting incremental er rors of P EMS m easurement w ere 

programmed into a computer model which employed Monte Carlo random sampling methods to 

simulate t he c ombined effects of a ll of t hese s ources of e rror on t he f inal m easured br akespecific value. An ideal reference NTE data set (see Section 2.5) for a given test event was run 

through the Model, and all the various errors were applied to that data set in a randomly chosen 

manner. Brake-specific PM emission values were then calculated for both the ideal and errorapplied data sets, which were compared to yield a final measurement error (see Appendix C and 

D). T he p rocess w as repeated t housands of t imes, w ith m any different i deal da ta s ets, t o 

generate a l arge, robust da ta s et w hich was ev aluated to determine a final s et of com bined 

measurement errors. These final errors, referred to in this report as deltas, were generated for the 

PM pollutant for each calculation method and three PEMS model units from each of the three 

manufacturers, for a final set of seven deltas; Methods 1-3 for the AVL PEMS unit, Methods 1-2 

for the Horiba PEMS unit and Methods 1-2 for the Sensors PEMS unit. A complete description 

of the Monte Carlo methodology and of the model is given in Section 2.5 of this report.

2.4 Calculation Methods 

Calculations m ust be pe rformed on t he r ecorded da ta t o de termine br ake-specific P M 

emission values in accordance with methods outlined in 40 C FR Part 10 65 Subparts G and J . 

The symbolic notation given in the formulas shown later in this section is fully described in 40 

CFR Part 1065 Subpart K [1]. 
REPORT 03.14936.12 5 of 174 

40 C FR P art 1065 a llows f or t he us e of a ny of t hree di fferent c alculation m ethods i n 

order to determine brake-specific emission values from in-use test data. The basic calculation of 

brake-specific emissions requires three main inputs as follows:

BS Emissions = 

Mass

Work =

Concentration x Flowrate

Power

The three calculation methods vary somewhat in the means used to determine either the 

Flow c omponent or t he W ork c omponent of t his c alculation. E ach of t he t hree m ethods i s 

summarized be low. B ecause e ach m ethod r elies on di fferent i nputs, i t i s pos sible that e ach 

method of c alculation w ill r eact di fferently t o va rious m easurement errors. T herefore, 

measurement allowances m ust be e xamined i ndependently f or e ach m ethod. H owever, 

according t o the T est P lan, see Appendix A and B , methodology, o nly on e of t he t hree 

calculation methods w ould be s elected to generate t he f inal m easurement a llowances. T he 

selection methodology i s outlined later in this introduction under the M easurement Allowance 

Generation section.

2.4.1 Calculation Method 1 – “Exhaust Flow-Torque-Speed” Method 

Calculation M ethod 1 i s a nalogous t o t he m ethod us ed b y most d ynamometer 

laboratories, and relies on direct input of both exhaust flow and torque. In the case of exhaust 

flow, this is the flow rate measured by the same form of exhaust flow meter. T he Sensors and 

AVL PEMS relied on the Sensors exhaust flow meter (EFM) while the Horiba PEMS had a tail 

pipe adapter (TPA) it employed for exhaust flow measurement. Work is not measured directly, 

but is instead calculated using ECM broadcast engine speed and ECM broadcast engine torque. 

While eng ine s peed is directly m easured by t he en gine E CM, ECM b roadcast t orque i s an 

estimate based on a variety of other parameters; torque cannot be directly verified during in-use 

testing. A simplified formula for this method is:

Method 1 = ∑ mass

∑ work
REPORT 03.14936.12 6 of 174 

The more complete formula used for Method 1 is as follows: 

For all PM PEMS: 

m PM is a flow weighted particulate matter exhaust concentration in g/mol 

( ) ( ) ( ) ∑

∑

=

=





 



 ⋅ ∆





 



  ∆ 

 



 



 





=⋅ N

i

i i

N

i

PM i

PM rpmSpeed mNT t

t

s

mol

n

mol

g m

ge kW hr

1

1

3600*1000*60

*14159.3*2**

* *

/



Where for AVL:

m PM

1

1

*

*

N

i i

i

PM N

i

i

g mol mPM n t

g mol s

m

mol mol

n t

s

=

=

        ∗ ∆            =         ∆      

∑

∑





is computed numerically as follows, 

It should be noted that calculation Method 1 is directly dependent on the accuracy of both 

the e xhaust f low m eter a nd t he t orque e stimation, a s w ell a s on t he m easurement of PM

concentration. 

2.4.2 Calculation Method 2 – “Exhaust and Fuel Flow-Torque-Speed” Method

This calculation is designated solely for in-use testing, and is designed to minimize the 

effect of errors r elated t o the a ccuracy o f t he ex haust f low m easurement. The M ethod 2 

calculation adjusts the exhaust flow measurement by a ratio of the CO2

Method 2 = ∑ mass

∑ � CO2 fuel

ECM fuel × Work�

-based fuel flow to the 

ECM r eported fuel f low. T his m eans t hat although t he f low m eter m ust be l inear, i t doe s not 

necessarily have to be accurate. In addition, Method 2 depends on t he ECM broadcast torque 

and s peed, a nd on t he r atio of f uel f low c alculated f rom t he c arbon ba lance us ing ga seous 

measurement over the fuel flow broadcast by the ECM. A simplified version of this method can 

be expressed as:
REPORT 03.14936.12 7 of 174 

 The more complete formula for Method 2 using PM as an example is:

For all PM PEMS: 

m PM is a flow weighted particulate matter exhaust concentration in g/mol 

( )

( ) ( ( ) ( ))

( ) ( )

1

6 2

2

* *

/

* *10 % % *10 *

*

* *2*3.14159

60*1000*3600

i

N

PM i

i

PM

ii i

C

fuel

fueli

i i

g mol m nt

mol s e g kW hr

mol n xTHC ppm xCO xCO t

M s

w g m

s

Speed rpm T N m

=

− −

      ∆       ⋅ =  

 

      ++ ∆    

         

⋅

   

∑ 



 1

N

i=

∑

Where for AVL:

m PM

1

1

*

*

N

i i

i

PM N

i

i

g mol mPM n t

g mol s

m

mol mol

n t

s

=

=

        ∗ ∆            =         ∆      

∑

∑





is computed numerically as follows, 

It should be noted that, as mentioned earlier, Method 2 i s not subject to accuracy errors 

for the exhaust flow measurement, although that measurement must still be linear for the method 

to function properly. 

2.4.3 Calculation Method 3 – “Fuel Flow-Torque-Speed” Method 

Method 3 doe s not us e di rect m easurement of e xhaust f low, but r elies on a carbon 

balance and ECM br oadcast f uel rate t o determine m ass. T he work term f or M ethod 3 i s 

determined identically t o t he w ork t erm f or M ethod 1; us ing t he E CM br oadcast va lues f or 

engine speed and torque to calculate work. Method 3 entirely circumvents the use of an exhaust 

flow meter, but for the HDIUT program, EPA must approve the use of Method 1 for a given test 

and manufacturer. A simplified version of Method 1 may be expressed as: 

Method 3 = 

∑ �mass × 

ECM fuel

CO2 fuel �

∑ Work
REPORT 03.14936.12 8 of 174 

 The more complete formula for Method 3 is:

For AVL Only:

( )

( ) ( ( ) ( ))

( ) ( )

6 2

1 2

1

/

* * *

*10 % % *10

* *2*3.14159*

60*1000*3600

i

PM

N ifue l

fuel

PM

C i i i

N

i i

i

e g kW hr

g m

g w s m t

mol M xTHC ppm xCO xCO

Speed rpm T N m t

− − =

=

⋅ =

              ∆   + +

 

  ⋅ ∆

   

∑

∑



Where:

( ( ))

( ) ( ( ) ( ))

( ) ( ( ) ( ))

6 2

1 2

6 2

1 2

/ *

* *

*10 % % *10

* *

*10 % % *10

i

i

N i ifue l

fuel

C i i i

PM

N ifue l

fuel

C i i i

g mPM g mol m w s t

M xTHC ppm xCO xCO

g m

mol g m

w s t

M xTHC ppm xCO xCO

− − =

− − =

          ∆

+ +

      =             ∆

+ +

 

∑

∑





It should be noted that Method 3 i s not subject to exhaust flow measurement accuracy 

errors, but also that this method is wholly dependent on ECM broadcast values for both mass and 

work determination. 

2.5 Reference NTE Events

The reference data set to which all the simulated errors were applied represented engine 

operations ove r a w ide r ange of laboratory NTE eve nts. P arameters i n the r eference da ta s et 

were scaled in order to exercise the model through a more appropriate range of parameters (i.e. 

concentrations, f lows, ambient c onditions, e tc.). In t his s caling pr ocess, c are w as t aken t o 

maintain the dynamic characteristics of the reference data set.

The Monte Carlo simulation model was run on a set of 141 reference NTE events that 

were used during the gaseous MA program [2]. O nly the events that have different speed and 

torque c ombinations w ere us ed; five e ngine m anufacturers pr ovided a t otal of 97 e vents; 10 

reference NTE events came from each of the three engines tested in the lab during the gaseous 

MA transient testing; and 14 e vents came from the pre-pilot CE-CERT data. Because no PM 

data exist for these reference events, the PM concentration used to calculate the Reference NTE 

PM emissions was developed by SwRI based on the Volvo engine used in this study. A simple 

model w as de veloped t o pr edict t he P M c oncentration ba sed on s peed a nd t orque us ing 80 

different s teady-state P M conc entrations t hat w ere m easured with the M SS us ing t he de sired
REPORT 03.14936.12 9 of 174 

exhaust b ypass c onfiguration. A dditional a djustment t o t he m odel us ing t he r ate of c hange i n 

torque was added to better predict short NTE transient events. The model was further adjusted to 

produce a br ake-specific e missions d istribution f or the r eference NTE e vents c entered a round 

0.02 g/hp-hr, as shown in Figure 1. 

 

FIGURE 1. METHOD 1 IDEAL BSPM VALUES FOR REFERENCE NTE EVENTS

NTE br ake-specific em issions r esults w ere calculated for PM us ing e ach of t he t hree 

agreed-upon NTE calculation methods. T he t hree di fferent BS emissions calculation methods 

referred to in this test plan are: 

1. Method No.1: Exhaust Flow-Torque-Speed Method 

2. Method No. 2: Exhaust and Fuel Flow-Torque-Speed Method 

3. Method No. 3: Fuel Flow-Torque-Speed Method 

Table 2 lists the number of NTE events obtained from each data source and the BSPM 

emissions calculated using Method 1. These emissions have been computed with no error values 

added to the input parameters. For this report, emissions with no errors added will be labeled the 

“ideal” emissions. In contrast, t he e missions w ith e rrors added t hrough t he M onte C arlo 

simulation will be labeled emissions “with errors.” 
REPORT 03.14936.12 10 of 174 

TABLE 2. REFERENCE NTE EVENTS AND IDEAL BSPM EMISSIONS

Source

Number of

NTE Events

BSPM

Min max

International 19 0.01146 0.03069

DDC 18 0.01303 0.03463

Caterpillar 20 0.01097 0.01789

Cummins 20 0.01561 0.03253

Volvo 20 0.01541 0.02701

Engine No. 1 10 0.01215 0.04066

Engine No. 2 10 0.01675 0.04872

Engine No. 3 10 0.01099 0.03669

When the ideal brake-specific emission values were calculated for the various reference 

NTE e vents, i t w as not ed t hat t hese i deal e mission va lues w ere f requently di fferent f rom one 

calculation method to another. While it was recognized that this was a realistic outcome, th e 

Steering C ommittee f elt tha t the se di screpancies mig ht int roduce a n unintended bias int o the 

results of the M onte C arlo simulation. T herefore, the S teering C ommittee di rected SwRI to 

adjust the NTE reference event data in order to align the brake-specific emission levels from all 

the calculation methods. 

The M ethod 1 r esult w as not c hanged, t herefore t orque, s peed, a nd e xhaust f low 

remained unchanged as well. The CO2 concentration was adjusted to make the Method 2 r esult 

equal to the one from Method 1. This was done by using a single multiplier on all CO2 values for 

the NTE event in qu estion. Lastly the fuel r ate values and alignments were adjusted to bring 

Method 3 in line with Method 1 and 2. 

The distribution of the ideal BSPM emissions data for the 141 reference NTE events was 

presented to the Steering Committee. It was noted that very few reference NTE events were at or 

below t he B SPM t hreshold ( 0.02 g /hp-hr). T hus, t he r eference N TE events were adjusted to 

produce more events with ideal BSPM values below 0.02 g/hp-hr. The original and revised ideal 

BSPM di stributions a re de picted i n Figure 1. N ote tha t the B SPM e mission data has va lues 

spread above and below the corresponding NTE threshold. 

Table 3 provides a summary of some descriptive statistics for the reference NTE data set 

for BSPM emissions.

TABLE 3. DESCRIPTIVE STATISTICS FOR BSPM EMISSIONS FOR 141 

REFERENCE NTE EVENTS

Descriptive Statistic BSPM g/hp-hr

Minimum 0.010974

Maximum 0.048717

Mean 0.020753

Median 0.019101

Standard Deviation 0.006932
REPORT 03.14936.12 11 of 174 

The pa rameter d ata pr ovided in each reference NTE eve nt w as on a s econd-by-second 

basis w ith a m inimum o f 30 s econds a nd a m aximum of 300 s econds. The i nput pa rameters 

required for the BSPM emissions calculation methods and the Monte Carlo simulation are listed 

in Table 4 . An E xcel f ile w ith a s pecific input f ormat s tructure w as us ed to standardize the 

format of the input files. Since the total hydrocarbons (THC) was selected as an input parameter, 

NMHC was computed as THC*0.98. 

TABLE 4. INPUT PARAMETERS FOR REFERENCE NTE EVENTS

Variable 

Number

Input Variable Units Description

1 NTE Event Number integer All reference NTE events must be identified by an NTE number (e.g., 

001).

2 NTE Source alphanumeric The source of the NTE event is the company, organization and/or lab 

that created the event data.

3 Engine Make alphanumeric Engine Make

4 Engine Model alphanumeric Engine Model

5 Engine Displacement L Engine Displacement (L)

6 Date mm/dd/yyyy The day the NTE event data was created (mm/dd/yyyy).

7 Time Stamp hh:mm:ss.s Time in seconds. Each reference NTE must contain second-bysecond data only.

8 Wet CO2 % CO2 (%)

9 Wet CO % CO (%)

10 Wet kNO ppm NO (ppm) with intake air-humidity correction

11 Wet kNO2 ppm NO2 (ppm) with intake air-humidity correction

12 Wet THC ppm THC (ppm)

13 Exhaust Flow Rate scfm Exhaust flow rate (scfm)

To compute the % of PEMS flowmeter maximum flowrate we will 

need to know what size flowmeter was used for each NTE event.

Enter either 3, 4, or 5 to represent the following flowmeters and 

maximum flow rates:

3 = 3 inch EFM with maximum flow rate = 600 scfm

4 = 4 inch EFM with maximum flow rate = 1100 scfm

5 = 5 inch EFM with maximum flow rate = 1700 scfm

15 Speed rpm Engine speed (rpm)

16 Low Speed, nlo rpm To compute the % of normalized speed we will need nlo and nhi for 

the engine computed as follows:

nlo (rpm) = lowest speed below max power at which 50% max power 

occurs

18 Fuel Rate L/sec Fuel rate (L/sec)

To compute the % of maximum fuel rate we will need the max fuel 

rate of the engine for each NTE event.

Max fuel rate (L/sec)

20 Derived Torque N·m Torque (N·m)

21 Peak Torque N·m To compute the % of maximum torque we will need the peak torque 

of the engine for each NTE event

Peak torque (N·m)

22 Flow-weighted Average 

PM Concentration for 

Methods 1 & 2

μg/mol Flow-weighted average PM concentration, flow-weighted by the 

exhaust flow. Values were calculated based on a predictive model 

developed using transient and steady-state experimental data.

23 Flow-weighted Average 

PM Concentration for 

Method 3

μg/mol Flow-weighted average PM concentration, flow-weighted by the 

exhaust flow. Values were calculated based on a predictive model 

developed using transient and steady-state experimental data.

19 Max Fuel Rate L/sec

nhi (rpm) = highest speed above max power at which 70% max 

power occurs

14 Flowmeter Diameter 3, 4, or 5 (inches)

17 High Speed, nhi rpm
REPORT 03.14936.12 12 of 174 

2.6 Error Surface Generation

During t he di scussions he ld at s everal S teering Committee m eetings, 33 error s urfaces 

were identified and considered for inclusion in the Monte Carlo simulation model. 25 of these 

error surfaces were the same surfaces used in the gaseous emissions in-use testing program. [2] 

Of t he r emaining ei ght er ror s urfaces, two were di scarded (Delta P M E MI/RFI and Delta P M 

Vibration). because the PM generator was not used during these tests (see Section 5) T he six 

new er ror s urfaces f or t he P M pr ogram w ere P M S teady-State ( SS), PM T ransient, PM 

Atmospheric Pressure, PM Ambient Temperature, Torque Engine Manufacturers and Fuel Rate 

Engine Manufacturers. This resulted in a final total of 31 error surfaces that were incorporated 

into the Model. These individual error surfaces encompassed a wide variety of error sources. In 

addition, all error surfaces distributions used in this program included a range of sampled data 

between the 1 st and 99 th percentile t o expand the r ange of s ample d ata i n the M onte C arlo 

simulation. T his w as done a t t he request o f t he S C s ince s ampling for t he gaseous p rogram 

covered only the range between the 5th and 95th. 

Table 5 lists the error surfaces examined during the study with the surfaces excluded by 

the S teering C ommittee de signated in italics. A ll r emaining on es w ere impl emented in the 

simulation model. Each error surface was assigned a number for easy identification.

For each of the measurement errors defined in Table 5, an error surface was created and

used i n t he M onte C arlo s imulation. E ach e rror s urface represented an a dditive e rror—or a

subtractive error if the sign was negative—relative to the reference parameter value to which it 

was applied. Figures 2 through 4 show an example of how these error surfaces were created for 

every measurement error. Details on t he c onstruction of e ach e rror s urface us ed i n t he 

simulation are provided in Section 0. The example illustrated in Figure 2 through 4 represents 

the e rror s urface for s teady-state bi as and precision PM concentration e rrors f or an i ndividual 

PEMS unit 

2.6.1 PEMS vs. Laboratory Nominal Results 

Figure 2 was constructed from raw data acquired from steady-state engine lab tests with 

the PEMS conducting repeat testing at various concentration levels (PM μg/mol). The plot pools 

all bias and precision errors for the PEMS tested for all steady-state modes. A nominal target of 

10 repeat m easurements of P M

Figure 2

was t aken on each PEMS unit f or each value of the 

corresponding a verage l ab P M values ( i.e., l ab nominal va lue). The 10 PEMS measurements

were pl otted a gainst t he c orresponding measurements using l aboratory equipment. S hown i n 

are t he 5 th, 50 th, a nd 95 th percentiles c orresponding t o t he di stribution of t hese 18 0 

observations (30 observations at each of the six concentration levels) using t he P EMS at each 

average P M concentration l evel ( note t hat t he distribution of da ta a t e ach P M level ma y not 

represent a normal distribution). S ince the 50th

Figure 2

percentiles do not lie on the dashed (diagonal) 

line of perfect agreement, the data suggest that there is a bi as error between the PEMS and lab 

results. In essence this graph summarizes the statistical distribution measured by the PEMS at 

each concentration level sampled. The example plot in shows only 6 discrete average 

PM concentration levels (ranging from 10-60 μg/mol). H owever, the actual number of discrete 

concentration levels was determined using the total number of operating conditions actually run 

for all the tests on t he engine. I n the section on Steady-State Repeat Engine Testing and Error 

Surfaces it is reported that 6 operating modes conditions from an initial number of 80 operating 
REPORT 03.14936.12 13 of 174 

conditions w ere s elected f or con struction of t he steady-state P M error s urface. From t hese 6 

operating modes several discrete PM concentration levels were defined which were used in the 

error surface plots for the Monte Carlo simulation.

TABLE 5. ERROR SURFACES FOR MONTE CARLO SIMULATION

Component # Test Source Error Surface Description

1. Delta PM 1 Engine Dyno Delta PM SS AVL, Horiba and Sensors

2 Engine Dyno Delta PM Transient AVL, Horiba and Sensors

3 Environ Delta PM EMI/RFI Deleted by Steering Committee

4 Environ Delta PM Atmospheric Pressure AVL

5 Environ Delta PM Ambient Temperature AVL

6 Environ Delta PM Vibration Deleted by Steering Committee

2. Delta CO 7 Engine Dyno Delta CO SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

10 Environ Delta CO Atmospheric Pressure Same as Gaseous Study 

11 Environ Delta CO Ambient Temperature Same as Gaseous Study 

3. Delta NMHC 13 Engine Dyno Delta NMHC SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

 NMHC = 0.98*THC 14 Engine Dyno Delta NMHC Transient Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

16 Environ Delta NMHC Atmospheric Pressure Same as Gaseous Study 

17 Environ Delta NMHC Ambient Temperature Same as Gaseous Study 

19 Environ Delta Ambient NMHC Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

4. Delta Exhaust Flow 20 Engine Dyno Delta Exhaust Flow SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

21 Engine Dyno Delta Exhaust Flow Transient Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

22 Engine Dyno Delta Exhaust Flow Pulsation Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

23 Engine Dyno Delta Exhaust Flow Swirl Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

25 Environ Delta Exhaust EMI/RFI Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

27 Environ Delta Exhaust Temperature Same as Gaseous Study 

28 Environ Delta Exhaust Pressure Same as Gaseous Study 

5. Delta Torque 29 Engine Dyno Delta Dynamic Torque Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

30 Engine Dyno Delta Torque DOE Testing

(Interacting Parameters Test)

Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

31 Engine Dyno Delta Torque Warm-up

(Interacting Parameters Test)

Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

32 Engine Dyno Delta Torque Humidity/Fuel

(Independent Parameters Test)

Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

34 Engine Dyno Delta Torque Interpolation Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

35 Engine Manuf Delta Torque Engine Manuf New

6. Delta Fuel Rate 42 Engine Manuf Delta Fuel Engine Manuf New

7. Delta Speed 43 Engine Dyno Delta Dynamic Speed Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

8. Delta Fuel Rate 44 Engine Dyno Delta Dynamic Fuel Rate Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

9. Delta CO2

45 Engine Dyno Delta CO2 SS Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

46 Engine Dyno Delta CO2 Transient Same as Gaseous Study but moved 5th% to 1st% 

and 95th% to 99th%

49 Environ Delta CO2 Ambient Temperature Same as Gaseous Study 

Measurement Error Surfaces and Deltas Used in BSPM Calculations
REPORT 03.14936.12 14 of 174 

FIGURE 2. ERROR SURFACE CONSTRUCTION: PEMS VS. LABORATORY 

RESULTS

2.6.2 (PEMS – Laboratory) Deltas vs. Lab 

Figure 3 illustrates the “error band” measured during testing. This plot was created by 

first subtracting the individual “lab nominal” PM value from the corresponding individual PEMS 

PM measurement for each test run. The sampling s ystem used to obtain the "lab nominal" or 

"lab reference" PM values is described in Section 4.2.1. The difference between the PEMS PM 

and the lab reference was defined as the “delta” error. Second, these “PEMS - Laboratory” delta 

errors w ere pool ed at each average l ab nom inal P M value t o obt ain t he 5 th, 50 th, a nd 95 th

percentile v alues di splayed i n Figure 3. Therefore, the pl ot r epresents t he ave rage P M l ab 

nominal at 6 discrete co ncentration levels ve rsus t he pe rcentiles of t he d elta er rors computed 

from the PEMS and laboratory individual test results. 

Error Surface for SS PM Concentration

18

26

38

45

56

69

13

24

36

42

53

64

8

19

33

39

51

61

0

10

20

30

40

50

60

70

80

0 10 20 30 40 50 60 70 80

PM μg/mole (lab,mean)

PM μg/mole (PEMS)

diagonal 95th percentile

50th percentile (median) 5th percentile
REPORT 03.14936.12 15 of 174 

FIGURE 3. ERROR SURFACE CONSTRUCTION: (PEMS - LAB) VS. LABORATORY 

RESULTS

2.6.3 Variability Index vs. (PEMS – Laboratory) Deltas and Lab Nominal 

This step normalized the plot in Figure 3 using what is called a “variability index (ic)”. 

This i ndex r epresented t he va lue r andomly dr awn b y t he M onte C arlo s imulation i n or der t o 

select a given error level. For the 5th and 95th percentile of the truncated normal it was allowed

to vary from –1 to +1, respectively. The likelihood of “ic” being any value between –1 through 

+1 w as s pecified b y a “ probability d ensity f unction ( PDF)” a ssigned t o i c. In the cas e of t his 

example, ic. was assumed to vary according to a standard normal (i.e, bell-shaped) distribution 

during the Monte Carlo simulations. T his was because it was believed that the distribution of 

PM errors due to steady-state bias and precision would be centered about the 50th

Figure 3

percentile of 

the full range of conditions measured. Each set of data for each lab “setpoint” average (i.e., lab 

nominal va lue) i n was nor malized b y aligning t he c orresponding 5 th

Figure 3

percentile er ror 

from with ic = -1, the 50th percentile error with ic = 0, and the 95th percentile error with 

ic = +1. These values were then plotted in Figure 4, where the y-axis is the variability index, the 

x-axis is the average lab nominal PM value, and the z-axis is the delta PM value. Notice that, 

when us ing t his nor malization a pproach, t he 5 th, 50 th, a nd 95 th

Error Surface for SS PM Concentration

8

6

8

5

6

9

3

4

6

2

3

4

-2

-1

3

-1

1 1

-4

-2

0

2

4

6

8

10

0 10 20 30 40 50 60 70

PM μg/mole (lab,mean)

PM μg/mole (PEMS) - PM μg/mole (lab)

95th percentile 50th percentile (median) 5th percentile

percentile va lues r emain 
REPORT 03.14936.12 16 of 174 

equivalent between Figure 3 and Figure 4. This development of the error surfaces from the lab 

data was the procedure used in the gaseous emissions in-use testing program. 

FIGURE 4. ERROR SURFACE CONSTRUCTION: ERROR AT VARIABILITY INDEX 

FOR 5TH AND 95TH PERCENTILES VS. LABORATORY RESULTS

For the PM measurement allowance program it was decided by the Steering Committee 

to expand the tails of the truncated normal distribution in the error surface formation in order to 

allow larger PM deltas to be sampled during the Monte Carlo simulation. Instead of truncating 

the lower tail at the 5th percentile, it was moved to the 1st percentile. Likewise on the upper tail, 

the 95 th percentile w as m oved t o t he 99 th percentile. S ince t he or iginal t runcated normal 

distribution was defined by a mean = 0 and a standard deviation = 0.60795, the resulting indices 

corresponding to the 1st and 99th

Figure 5

percentiles are -1.4143 and +1.1413, respectively, as illustrated 

in . 

8 6 8 6 9

3 4 6 2 3 4

-2 -1 3 -1 1 1

-1.5

-1

-0.5

0

0.5

1

1.5

0 10 20 30 40 50 60 70

ic_SS_PM

PM μg/mole (lab,mean)

Error Surface for SS PM Concentration

95th percentile 50th percentile (median) 5th percentile

Error Surface: z-axis = ΔSS_PM_μg/mole

5
REPORT 03.14936.12 17 of 174 

FIGURE 5. TRUNCATED NORMAL DISTRIBUTION PERCENTILES

Finally, the 5th and 95th percentile (PEMS – LAB) delta values at each lab nominal value 

were us ed t o c ompute t he c orresponding va lues of t he t runcated nor mal f or t he 1 st and 99 th

percentiles. To redefine the error surface delta at the 1st percentile, the standard deviation for the 

normal distribution below the 50th

Standard Deviation 1st = 

Delta Value50th − Delta Value5th

1.6449

 percentile is defined as follows: 

 Using the mean = 50th percentile delta and the standard deviation computed above, the 1st

percentile can be found using the Excel NORMINV function: NORMINV(0.01,mean,standard 

deviation1st). Similarly, to redefine the error surface delta at the 99th percentile, the standard 

deviation for the normal distribution above the 50th

Standard Deviation 99th = 

Delta Value95th − Delta Value50th

1.6449

 percentile is defined as follows: 

Using the mean = 50th percentile delta and the standard deviation computed above, the 

99th percentile c an be f ound us ing t he E xcel N ORMINV f unction: 

NORMINV(0.99,mean,standard deviation99th). Taking the data for the error surface in Figure 4

and redefining it to include the 1st and 99th

Figure 6

percentile truncated normal values results in the error 

surface displayed in . All the error surfaces carried over from the gaseous program that 

were sampled using a truncated normal were redefined at the 1st and 99th percentiles. Those error 

surfaces that were sampled using the uniform distribution remained unchanged. Error surfaces 

such as the one presented in Figure 6 a re the error deltas the Monte Carlo simulation program 

used during calculation of the BSPM emissions “with errors”. 

-3 -2 -1 0 1 2 3

Monte Carlo Truncated Normal Sampling Distribution

N(0,0.60795)

-1.4143

Area= 0.05

Area = 0.01

-1.0 1.0

5th 95th

1.4143

Percentiles→ 1st 99th
REPORT 03.14936.12 18 of 174 

FIGURE 6. ERROR SURFACE CONSTRUCTION: ERROR AT VARIABILITY INDEX 

FOR 1ST AND 99TH PERCENTILES VS. LABORATORY RESULTS

2.7 Error Surface Sampling and Interpolation

The e rror m odel us ed t wo di fferent PDF to sample t he er ror s urfaces, depending upon 

which e xperimental pa rameter the s urface r epresented. T o sample er ror s urfaces t hat w ere 

generated from the l ab test results (Section on Engine Dynamometer Laboratory Testing), and 

the applicable environmental t est r esults, t he m odel us ed a t runcated s tandard nor mal PD F 

because these tests were designed to evenly cover the full, but finite, range of engine operation 

and a mbient c onditions. T o s ample error s urfaces t hat w ere generated from t he pr essure and 

temperature environmental test results (Section on Environmental Chamber Testing), the model 

used a uni form P DF b ecause t hese t ests were already d esigned to cove r t he t ypical range an d 

frequency of t he respective c onditions. Both of t hese s ampling di stributions a re depicted i n 

Figure 7. 

10.1 6.8 8.8 7.2 11.1

3 4 6 2 3 4

-4.1 -3.1 1.8 -2.2 0.2 -0.2

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 10 20 30 40 50 60 70

ic_SS_PM

PM μg/mole (lab,mean)

Error Surface for SS PM Concentration

99th percentile 50th percentile (median) 1st percentile

Error Surface: z-axis = ΔSS_PM_μg/mole

6.2
REPORT 03.14936.12 19 of 174 

FIGURE 7. TRUNCATED STANDARD NORMAL AT 1ST AND 99TH PERCENTILES 

AND UNIFORM PROBABILITY DENSITY FUNCTIONS 

When using the truncated standard normal PDF (see Figure 7), the Monte Carlo model 

sampled normal deviates that ranged between -1.4143 and +1.4143. These were used as the ic

values de fined i n t he s ection on Error Surfaces. Similarly, t he pr essure a nd t emperature 

environmental tests used a uniform PDF to sample test time, from which calculated errors were 

used. All temperature error surfaces related to the four emissions were sampled uniformly from 

1 to 1080 minutes while the error surfaces related to the pressure were sampled uniformly from 1 

to 720 m inutes. E xhaust flow error surface for temperature was sampled uniformly from 1 t o 

478 minutes while the exhaust flow for pressure was sampled uniformly from 1 to 360 minutes. 

The errors from all the other tests were aligned with the truncated standard normal PDF such that 

each of the 50th percentile error values at each of the tested signal magnitudes was centered at the 

median (i.e., 0 value) of the PDF, and the 1st and 99th percentile error values at each of the tested 

signal ma gnitudes w ere a ligned with the e xtreme ne gative ( ic = -1.4143) a nd pos itive (ic = 

+1.4143) edges of the PDF, respectively.

Each error s urface was sampled along i ts i c axis ( y-axis) onc e p er trial for a reference 

NTE event simulation. Hence, every error surface had a s eparate randomly selected ic for each 

trial. S ince e ach reference NTE event contained s econd-by-second parameter data, ex cept for 

PM for the Sensors PPMD and Horiba TRPM PEMS, the error surface was sampled at a given ic 

on the y-axis and at the several selected parameter values on the x-axis that corresponded to each 

second of t he r eference N TE eve nt. The s ampled error value was determined f or t he given 

second and parameter al ong t he error ax is ( z-axis) a t the int ersection of the ic

Probability Density Functions for Sampling Error Surfaces Once Per NTE Event 

-1.41

0.00

1.41

5 4 3 2 1 0

Relative Probability

i c

Lab Tests, Normal, SD=0.60795, truncate @ -1.4143 & 1.4143

Environmental Tests, Uniform

Note: A non-truncated normal 

distribution with SD=0.60795 has P 

values of 0.01 and 0.99 at i c =-1.4143 and 

ic=+1,4143 respectively.

value a nd t he 

parameter value from the reference NTE event. This was accomplished by taking each second in 

the r eference N TE event and finding t he t wo adjacent x -axis va lues f rom t he er ror s urface 
REPORT 03.14936.12 20 of 174 

between which t o l inearly i nterpolate t o obt ain the e rror s urface x -value. Each s econd i n the 

reference NTE event was linearly interpolated with the same i c value for a pa rticular trial at the 

error surface x-value. If any of the sampled lab nominal values (PM, NMHC, CO, Speed, Fuel 

Rate, etc.) ex ceeded the uppe r or l ower l imits o f t he p arameter er ror s urface, t he value of t he 

closest endpoint of the error surface was assigned to them.

Figure 8 depicts an example of the error surface sampling using a steady-state PM error 

surface c ontaining 10 l ab nom inal P M x-axis v alues. F or thi s pa rticular tr ial, the r andomly 

selected ic is -0.5. The example reference NTE event is noted by the symbol ‘*’ and it plotted at 

ic = -0.5 for each second in the NTE event. 

FIGURE 8. STEADY-STATE PM ERROR SURFACE FOR AVL WITH EXAMPLE 

SAMPLING FOR A REFERENCE NTE EVENT

2.8 Brake-Specific Emissions Calculations

Errors from Sections 4, 5, and 6 were combined by adding all of the sampled errors once 

per trial for each reference NTE event simulation. For example, in order to assess the errors in 

PM concentration by calculation Method #1, several error surfaces were sampled and added to 

the corresponding parameter in the Method #1 calculation and the resulting BSPM “with errors” 

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 50 100 150 200 250 300 350 400 450

ic_SS_AVL

PM μg/mol (lab,nom)

SS Error Surface for AVL PM Concentration

99th percentile 50th percentile (median) 1st percentile

ic = -0.5 NTE Event

Error Surface: z-axis = ΔSS_PM_AVL_μgmol
REPORT 03.14936.12 21 of 174 

was computed. The errors used in this calculation for the Horiba and Sensors are the following 

(note that the corresponding error surface numbers are provided in the subscripts):

PM μg/mol ‘with errors’ = PM μg/mol_reference + ∆ PM μg/mol_1

∆ PM 

+ 

μg/mol _2 

Exhaust Flow % ‘with errors’ = Exhaust Flow % reference 

∆ Exhaust Flow 

+ 

% 20 ∆ Exhaust Flow_% 21 

∆ Exhaust Flow 

+ 

% 22 + ∆ Exhaust Flow % 23 

∆ Exhaust Flow 

+ 

% 25 + ∆ Exhaust Flow % 27 

∆ Exhaust Flow 

+ 

% 28 

Torque % ‘with errors’ = Torque % reference 

∆ Torque 

+ 

% 29 + ∆ Torque % 30 

∆ Torque 

+ 

% 31 + ∆ Torque % 32 

∆ Torque 

+ 

% 34 + ∆ Torque % 35 

Speed % ‘with errors’ = Speed % reference + ∆ Speed 

 

% 43 

where,

∆ 1,2

∆

= PM concentration errors due to steady-state and transient errors,

20,21

∆

= exhaust flow errors due to steady-state and transient errors,

22,23

∆

= exhaust flow errors due to pulsation and swirl, 

25

∆

= exhaust flow errors due to ambient temperature, 

27,28

∆

 = exhaust flow errors due to temperature and pressure, 

29

∆

 = torque errors due to dynamic torque, 

30,31

∆

 = torque errors due to DOE and warm-up, 

32

∆

 = torque errors due to interacting parameters humidity and fuel,

34,35

∆

 = torque errors due to interpolation and engine manufacturers, 

43 = speed errors due to dynamic speed

Using the formulas for the calculation methods, the BSPM for Method #1 was computed 

without errors (“ideal”) and then with all the errors applied as outlined above. Table 6 lists all 

error surfaces used by each calculation method for the PM emissions. 
REPORT 03.14936.12 22 of 174 

TABLE 6. ERROR SURFACES USED FOR COMPUTING BRAKE-SPECIFIC PM 

EMISSIONS BY THREE CALCULATION METHODS

2.9 Convergence and Number of Trials

Since t he T est P lan did not i nclude a p rovision for conv ergence criteria, the S teering 

Committee w as t asked t o de velop a c onvergence m ethod. T he m ain g oal w as t o de fine how 

many s imulation trials at a g iven reference N TE eve nt w ere r equired t o estimate the 95 th

percentile BSPM e mission differences w ith a given precision. A lthough the C rystal Ball 

software contained precision control options, the method used to compute a confidence interval 

on pe rcentiles w as b ased on a n a nalytical boot strapping m ethod w hich w as not a dequately 

documented. T hus, a n independent c onvergence m ethod w as pr oposed a nd a ccepted b y t he 

Steering Committee.

Component # Error Surface Method 1 Method 2 Method 3

1. Delta PM 1 Delta PM SS   

2 Delta PM Transient   

4 Delta PM Atmospheric Pressure   

5 Delta PM Ambient Temperature   

2. Delta CO 7 Delta CO SS  

10 Delta CO Atmospheric Pressure  

11 Delta CO Ambient Temperature  

3. Delta NMHC 13 Delta NMHC SS  

 NMHC = 0.98*THC 14 Delta NMHC Transient  

16 Delta NMHC Atmospheric Pressure  

17 Delta NMHC Ambient Temperature  

19 Delta Ambient NMHC  

4. Delta Exhaust Flow 20 Delta Exhaust Flow SS  

21 Delta Exhaust Flow Transient  

22 Delta Exhaust Flow Pulsation  

23 Delta Exhaust Flow Swirl  

25 Delta Exhaust EMI/RFI  

27 Delta Exhaust Temperature  

28 Delta Exhaust Pressure  

5. Delta Torque 29 Delta Dynamic Torque  

30 Delta Torque DOE Testing

(Interacting Parameters Test)  

31 Delta Torque Warm-up

(Interacting Parameters Test)  

32 Delta Torque Humidity/Fuel

(Independent Parameters Test)  

34 Delta Torque Interpolation  

35 Delta Torque Engine Manuf  

6. Delta Fuel Rate 42 Delta Fuel Engine Manuf 

7. Delta Speed 43 Delta Dynamic Speed  

8. Delta Fuel Rate 44 Delta Dynamic Fuel Rate 

9. Delta CO2 45 Delta CO2 SS  

46 Delta CO2 Transient  

49 Delta CO2 Ambient Temperature  
REPORT 03.14936.12 23 of 174 

A nonparametric statistical technique [3] was proposed which defined a 90% confidence 

interval f or t he 95 th percentile of t he BSPM em issions di fferences f or a n individual r eference 

NTE simulation. I f the width of the 90% confidence interval was less than 1% of the BSPM 

emissions t hreshold, t hen c onvergence w as m et. T he f ollowing s teps d efine t he conv ergence 

method: 

1. Run the Monte Carlo simulation for N trials.

2. Order the BS emissions differences from smallest to largest.

3. Identify the trial number at the lower end of the 90% confidence interval

nlower = N − *05.0*95.0645.1*95.0 N

4. Identify the trial number at the upper end of the 90% confidence interval 

nupper = N + *05.0*95.0645.1*95.0 N

5. Compute (BSPM difference value at nupper) – (BSPM difference value at nlower

6. If the result in (5) < 1% of the BSPM emissions NTE threshold then convergence is met. 

).

7. The BSPM threshold was defined as 0.02 g/hp-hr. Thus, 1% of the threshold was 0.0002 

g/hp-hr.

The Steering Committee a greed to the proposed c onvergence c riteria outlined a bove. 

During the initial simulation runs for 20 reference NTE events, convergence was not met at the 1 

percent criteria level until 60,000 t rials were run. T his only applied to the AVL PEMS unit at 

each of the three calculation methods. The Horiba and Sensors units only reached convergence 

at t he 1 percent cr iteria f or appr oximately h alf of t he 20 reference N TE eve nts s imulations. 

Upon e xamination of t he di stributions of t he de lta BSPM e missions ge nerated f rom t he 

simulations, some of the distributions were positively skewed which would make convergence 

very difficult at the 1 percent level. This information was presented to the Steering Committee 

wherein a decision was made to relax the convergence level to 2 percent or higher, depending on 

the outcomes of the simulations. 

 In s ummary, t he 141 r eference N TE e vents w ere r un a t 40,000 t rials a nd c onvergence 

was checked. If the width of the confidence interval on the 95th percentile delta BSPM emission 

was approaching 2 p ercent of the threshold, then the simulation was continued for up to 65,000 

trials.

2.10 Simulation Output

During t he s imulation of a r eference NTE e vent, differences b etween the B SPM 

emissions “with errors” and the ideal BSPM emissions were obtained by each of the three PEMS 

model uni ts a nd e ach of t he t hree applicable c alculation m ethods. T hese differences were

computed thousands of times (once per trial) until the model converged. Then the 95th percentile 

difference value was determined for each reference NTE event’s distributions of BS differences 

for the PM emissions for all three PEMS units and applicable calculation methods. 

The output from the Crystal Ball simulation for each reference NTE event was saved in 

two separate E xcel f iles: an EXTRACT and a REPORT f ile. T he E XTRACT f ile cont ained 

descriptive s tatistics on all di fferences c omputed f or B SPM e missions b y all thr ee c alculation 

methods, pe rcentiles ( 0%, 5% , 10% ,...95%, 100% ) of t he di fferences i n B SPM e missions, 
REPORT 03.14936.12 24 of 174 

sensitivity data for all error surfaces, and differences in BSPM emissions computed at each trial 

in the simulation.

The REPORT file contained a summary of the differences in the BSPM emissions for all 

three PEMS units and applicable calculation methods including descriptive statistics, the number 

of trials, a frequency histogram of the differences in BSPM emissions, and percentiles (0%, 5%, 

10%,...95%, 100% ) of the di fferences i n B SPM e missions. A lso i ncluded w ere de scriptive 

statistics on each i c distribution sampled for each error surface. Lastly, sensitivity charts for the 

differences in BSPM emissions for the three PEMS unit and applicable calculation methods were 

stored. T hese c harts pr ovided i nformation on how m uch e ach e rror s urface i nfluenced t he 

differences computed between the BSPM em issions “w ith errors” a nd the i deal BSPM 

emissions. A m ore de tailed de scription of t he C rystal Ball out put f iles c an be f ound i n 

Appendix C. 

2.11 Step-by-Step Simulation Example

 In o rder t o c larify t he s imulation pr ocess, the f ollowing s tep-by-step summary is 

provided. This example assumes that a single reference NTE event was simulated for the BSPM 

difference computations. Figure 9 provides an overview of the simulation process. 

FIGURE 9. OVERVIEW OF MONTE CARLO SIMULATION FOR BSPM
REPORT 03.14936.12 25 of 174 

Step 1 - Enter the reference NTE input parameters into the Monte Carlo (MC) simulation 

model. T hese i nclude t he e missions c oncentrations, e xhaust f low, t orque, s peed and f uel r ate 

data used in all three calculation methods. 

Step 2 - Compute the “ideal” B SPM b y al l three P EMS m odel uni t and a pplicable 

calculation methods from the reference NTE event.

Step 3 - Set-up the M onte C arlo simulation parameters in Crystal B all. An Excel 

spreadsheet m odel was de veloped for us e w ith Oracle® Crystal Ball M C s oftware for e rror 

analysis of brake-specific emissions. Crystal B all is graphically-oriented f orecasting a nd 

simulation software that runs on M icrosoft Windows and Excel. The simulations run in this 

program us ed C rystal Ball V ersion 11.1.1 a nd were run on P Cs c onfigured w ith a P entium 4 

CPU, 3.39 G Hz, 3.50 G B R AM, 232 GB ha rd dr ive a nd W indows X P ope rating s ystem. 

Microsoft Excel 2003 SP was the spreadsheet software.

The options exercised in running Crystal Ball included the following: 

• Number of trials = 40,000 

o If convergence was not met at the 2% criteria then # trials = 65,000 

• Monte Carlo sampling method with random initial seeds 

• Normal speed run mode 

• Suppress chart windows (fastest run time) 

The Excel spreadsheet i s i n a m odular s tructure f ollowing t he s pecified m odel out line, a nd i t 

makes pr ovisions f or t he t hree i dentified c alculation m odules. Input c ells t o the m odel a re 

clearly id entified to facilitate an y revisions t hat may become ne cessary f or us ers who w ant to 

exercise the model with other Monte Carlo software such as @Risk or newer versions of Crystal 

Ball. The spreadsheet was tested with controlled test cases of simplified input distributions with 

the C rystal Ball a dd-on to confirm c orrect model impl ementation in accordance with this te st 

plan. A t least one typical analysis was run as an additional confirmation, and two independent 

checks were made on the ideal emissions by other SwRI staff. A complete description of the 

spreadsheet computations is contained in Appendix D. 

Step 4 - Execute a single MC trial by randomly generating a separate i c for each error 

surface used in the three calculations.

Step 5 - For each second in the reference NTE event, interpolate the ∆ error for all error 

surfaces at the input parameter values and the randomly generated ic. Figure 10 illustrates all the 

error s urfaces a vailable a nd w here t he corresponding ∆ errors are a dded. T he num bers i n 

parentheses represent the error surface number in the Monte Carlo simulation. 
REPORT 03.14936.12 26 of 174 

FIGURE 10. ERROR SURFACES INCLUDED IN MONTE CARLO SIMULATION

Step 6 - Compute one BSPM “with errors” for the given MC trial b y adding all the ∆

error values to the reference NTE data and then calculating the BSPM by all three PEMS units 

and applicable calculation methods.

Step 7 - Compute BSPM difference for the current trial:

 BSPM emission “with errors” – “Ideal” BSPM emission

Step 8 - Repeat Steps 4-7 until the number of trials is met.

Step 9 - Check the di fferences i n BSPM f or all t hree P EMS uni ts a nd applicable 

calculation methods to be certain that the convergence criteria are met. If convergence is met for 

all three calculation methods, continue to Step 10. Otherwise, return to Step 4 and run the Monte 

Carlo simulation for an additional 25,000 trials until the total number of trials is 65,000. 

Step 10 - Select the 95th percentile from the distribution of BSPM differences for each of 

the three PEMS units and applicable calculation methods. S tore the ideal BSPM and the 95th

percentile BSPM differences for computing the measurement allowance.

Step 11 - Repeat Steps 1-10 for each reference NTE event.

Environ 

chambers

Engine

dyno

Environ 

chambers

Engine 

dyno

PM

∆PMSS_ppm (1)

∆PMTR_ppm (2)

∆PMpress_ppm (4)

∆PMtemp_ppm (5)

Εxhflow

∆ExhflowSS (20)

∆ExhflowTR (21)

∆Exhflowpulsation (22)

∆Exhflowswirl (23)

∆ExhflowEMI/RFI (25)

∆Exhflowtemp (27)

∆Exhflowpressure (28)

Speed ∆SpeedDynamic (43)

∆Fuel RateDynamic Fuel Rate (44)

Torque

∆TorqueDynamic (29)

∆Torquewarmup (31)

∆Torquehumidity (32)

∆Torqueengine-manuf (35)

Engine dyno

∆Fuel Rateengine-manuf (42)

Engine 

dyno

CO

∆COSS_ppm (7)

∆COpress_ppm (10)

∆COtemp_ppm (11)

Engine 

dyno

NMHC

∆NMHCSS_ppm (13)

∆NMHCTR_ppm (14)

∆NMHCpress_ppm (16)

∆NMHCtemp_ppm (17)

∆NMHCambient (19) Engine 

dyno

CO2

∆CO2SS_ppm (45)

∆CO2TR_ppm (46)

∆CO2temp_ppm (49)

Environ 

chambers

Environ 

chambers

∆TorqueDOE (30)

∆Torqueinterpolation(34)
REPORT 03.14936.12 27 of 174 

2.12 Measurement Allowance Generation

The generation of a set of measurement allowances represented the final outcome of this 

program. T he Test Plan provided a methodology by which all of the data from the millions of 

Model simulation runs would be collected and analyzed statistically, in order to generate a set of 

potential m easurement a llowances f or each P EMS m odel uni t, one f or e ach of t he c alculation 

methods. T he Test Plan then outlined a specific method b y which the final set of allowances 

would be chosen from among deltas generated for each of the three calculation methods. T he 

assumption made by the Test Plan, was that the final outcome of all previous efforts would be a 

set of validated pot ential measurement al lowance va lues f or e ach PEMS uni t. E ach potential 

allowance w as ex pressed as a pe rcentage of i ts as sociated BSPM N TE t hreshold. T hese 

measurement allowances w ere computed b y a r egression m ethod or a m edian m ethod a s 

described below.

2.12.1 Regression Method

This method involved determining the correlation between the 95th percentile differences

versus the ideal emission values for the reference NTE dataset. For each combination of PEMS 

units and calculation method, a least squares linear regression of the 95th percentile differences 

versus the ideal emissions results was computed. If the R2 value from the regression model was 

greater than 0.85 and the SEE (standard error of the estimate or root-mean-squared-error) was 

less t han 5 percent of t he m edian ideal BS e missions, t hen t he linear r egression equation was 

used to determine the measurement allowance for that PEMS unit and calculation method. To 

determine the measurement allowance, the NTE threshold was used to predict the measurement 

allowance f rom t he r egression model. T he m easurement al lowance was t hen expressed as a

percentage of the NTE BSPM threshold value (0.02 g/hp-hr).

2.12.2 Median Method 

If t he l inear r egression di d not pa ss t he a forementioned criteria for t he R 2 and SEE

statistics, then the median value of the 95th percentile differences from the 141 reference NTE 

events was us ed as t he s ingle m easurement allowance f or a c ombination of e missions a nd 

calculation method. The measurement allowance was then expressed as a percentage of the NTE 

threshold value. 

After a ll 95 th percentile distributions w ere e valuated, there w ere s even measurement 

allowances c orresponding t o t he combinations of t he t hree PEMS uni ts and t he applicable

calculation methods. 

Next, the calculation method with the minimum normalized PM value will be chosen and 

the corresponding normalized PM value will be selected as the best measurement allowance for 

PM, assuming it validates. This PM measurement allowance would be the very last value added 

to the act ual br ake-specific N TE PM threshold f or a g iven e ngine, ba sed o n actual f amily 

emissions limit, mileage, model year, etc. N ote tha t if a ny m easurement al lowance w as

determined to have a va lue l ess t han zero, then that m easurement al lowance w as s et equa l t o 

zero.
REPORT 03.14936.12 28 of 174 

The BSPM NTE threshold used for this program was 0.02 g/hp-hr. This NTE threshold 

was determined by EPA and approved by the Steering Committee during the generation of the 

Test Plan.

 These threshold values are of critical importance to the program, as they provide the basis 

for the scaling of measurement allowances, the assessment of model convergence, and a variety 

of other calculations performed during this program. T he general philosophy of the Test Plan 

was to determine measurement allowances based on errors at these emission levels, especially in 

the case of any errors that scaled with emission level.

The a nticipated out come f rom t he m odel r uns, a nalysis, a nd va lidation efforts c an b e 

represented as a table similar to the one shown in Table 7. The table illustrates both the model 

outcome, a nd t he p rocess f or s electing t he f inal m easurement allowance va lues for t he AVL 

PEMS.

TABLE 7. EXAMPLE OF SELECTION OF MEASUREMENT ALLOWANCE AT 0.02 

G/HP-HR NTE THRESHOLD FOR THE AVL PEMS

Allowance at Respective NTE threshold (%)

Calc. Method 

Method 1

Exhaust Flow 

Torque-Speed

Method 2

Exhaust and Fuel 

Flow Torque-Speed

Method 3

Fuel Flow TorqueSpeed

BSPM 38 % 18 % 20 % 

Selected Method  Exhaust and Fuel Flow Torque-Speed Method

The i ntent of t he f inal s election pr ocess was t o choose the s mallest of the thr ee 

normalized PM values for the final measurement allowance. At that point, the percentages given 

for the chosen calculation method would be applied to the BSPM NTE threshold value in order 

to generate the final additive, brake-specific measurement allowances.

An implicit assumption of the process, as described in the Test Plan, was that the values 

produced by the model for all three calculation methods would be successfully validated. In the 

event that this did not occur, it would be necessary for the Steering Committee to determine a 

valid alternate course of action, in order to determine the final measurement allowance values.

The final model run and the selection and generation of measurement allowances are described 

fully in Section 0 of this r eport, i ncluding t he f inal a llowances a pproved b y t he S teering 

Committee.

2.13 Model Validation 

For reasons discussed earlier, the measurement allowances were generated using a Monte 

Carlo computer model. As with all simulations, it is vital that such a model be validated through 

comparison with real experimental data. In this case, the Measurement Allowance model needed 

to be va lidated against a da ta s et generated through actual i n-use f ield t esting. B ecause t he 

model generates an incremental error in comparison to a Laboratory Reference, a suitable in-use 

reference measurement was needed for comparison to the PEMS measurements. T he Steering 
REPORT 03.14936.12 29 of 174 

Committee de termined that the C E-CERT M obile E mission L aboratory, op erated b y t he 

University of C alifornia-Riverside, w ould be a n a ppropriate r eference f or va lidation of t he 

model-based in-field testing.

To ensure that the validation was not disturbed by some inherent bias between the SwRI 

Reference Laboratory and the CE-CERT MEL validation reference, a correlation exercise was 

performed between the two laboratories, prior to the start of on-road validation efforts. The CECERT MEL was brought to SwRI’s laboratory facilities in San Antonio, Texas, and a side-byside correlation test was run. During this test, exhaust from the same test engine was alternately 

routed t o t he m easurements s ystems of bot h S wRI and C E-CERT. This w as don e r epeatedly 

over the course of three days of testing. The data was then supplied to the Steering Committee, 

in order to allow f or a determination to be ma de tha t c orrelation between the f acilities w as 

acceptable for the purposes of validation of the model. 

After the correlation exercise was completed, a 2007 test truck with a Cummins engine 

was procured by CE-CERT for use in this validation exercise. I n addition, two Sensors PEMS 

used at SwRI during the program were also delivered to CE-CERT. A third PEMS unit of the 

same t ype w as pr ovided b y S ensors. The s teering c ommittee allowed Sensors t o pr ovide a 

similar model with some small hardware upgrades for testing. CE-CERT then conducted a series 

of on-road test runs over various driving routes in California, which were designed to take the 

test truck through a wide range o f environmental and ambient conditions. D uring these tests, 

simultaneous m easurements w ere m ade w ith the P EMS and the M EL in order t o generate a 

validation data set. This formed the primary validation set for the model.

Because t he C E-CERT M EL doe s not r eadily i ncorporate a m eans of di rect t orque 

measurement on a vehicle, the on-road validation data set could not be used to validate model 

errors associated with broadcast torque. 

The di fference be tween t he P EMS r esults and t he CE-CERT trailer r esults w ill be 

compared to the measurement allowance limits predicted by the Monte Carlo Model and defined 

by the LOESS fit. 

Validation will be based on the following procedure. For each reference NTE event, the 

Monte C arlo m odel w ill be us ed to generate t he 5 th and 95 th percentiles of t he s imulated 

distribution of t he br ake-specific P M em ission differences. In o rder t o obt ain s imulations 

representing similar conditions to those obtained on-road, some error surfaces may ne ed to be 

suppressed in the simulations since not all of them may be applicable to the on-road conditions. 

The c hoice of w hich error s urfaces t o s uppress w ould ne ed t o b e made b y t he S teering 

Committee. 

Next, the 5 th and 95 th delta pe rcentiles obt ained f rom t he a bove s imulations w ill be 

separately fit to a line or curve using two chosen methods: a linear regression procedure and a 

local regression (loess) technique [4]. Depending on w hich of the resulting two fits is best for 

each set of data (i.e., either for the 5th percentile deltas or the 95th percentile deltas), the resulting 

line or curve will be used as one of the lower or upper limits for the on-road data.
REPORT 03.14936.12 30 of 174 

To determine the best fit for a given set of delta percentiles (i.e., 5th or 95th), a simple 

regression line initially will be fit to the data. If a least squares linear regression of the 5th or 95th

percentile deltas versus the ideal PM emission has an r

2 greater than 0.85 and an SEE less than 5 

percent of the median ideal PM emissions, then the regression line will be used. If this set of 

criteria is not met, then a loess fit will be used. Since a loess regression requires the selection of 

a smoothing parameter [5] to smooth the data, the chosen smoothness parameter should balance 

the residual sum of squares against the smoothness of the fit. 

The on -road delta e rrors, obtained from t he results of col lecting d ata on s everal N TE 

events during on-road operations, will be plotted on a graph containing the 5th and 95th percentile 

delta limits determined from the regression fits chosen above. The graph will consist of a plot of 

delta PM versus ideal PM. The number of on-road points outside these limits will be determined 

and expressed as a percentage of the total number on on-road data points. If this number does not 

exceed 10% of the total number o f on-road d ata, the simulation data will be considered to be 

valid. 

 An example of a validation plot is given in Figure 11 and Figure 12. T he plots shown 

correspond to gaseous emissions concentration data that were collected in the prior PEMS study. 

Figure 11 contains the 5th and 95th validation limits for NOX

Figure 12

data determined by fitting a linear 

regression model to the s imulated data for bot h limits. contains t he 5 th and 95 th

validation limits for NOX data determined by fitting a loess model to the simulated data for both 

limits. 

FIGURE 11. LINEAR REGRESSION FIT TO 5TH AND 95TH PERCENTILE DELTAS
REPORT 03.14936.12 31 of 174 

FIGURE 12. LOESS REGRESSION FIT TO 5TH AND 95TH PERCENTILE DELTAS

Validation of the model was assessed independently for the PM pollutant for each of the 

three P EMS m odel uni ts, a nd f or e ach of t he applicable t hree c alculation m ethods. A full 

description of the validation efforts, including the data analysis methodology and the results of 

PM validation for each PEMS unit by all three calculation methods is given in Section 2.4, with

the exception of the CE-CERT on road validation testing. 
REPORT 03.14936.12 32 of 174 

3.0 PART 1065 PEMS AND LABORATORY AUDIT

Prior to the start of official testing both the laboratory system and each PEMS underwent 

an extensive audit in accordance with CFR Part 1065. Table 8 summarizes the audits that were 

performed on each type of system along with the CFR reference for each of the verifications. 

TABLE 8. LINEARITY VERIFICATION RESULTS FOR INTAKE AIR FLOW AND 

FUEL FLOW

Fuel Flow

Description |xmin(a1-1)+ao| a1 SEE R2

Measured -0.02 1.00 0.14 1.000

Criteria 0.55 0.98-1.02 1.10 0.990

Pass / Fail Pass Pass Pass Pass

Intake Air Flow

Description |xmin(a1-1)+ao| a1 SEE R2

Measured 0.00 1.00 1.22 1.000

Criteria 15.68 0.98-1.02 31.36 0.990

Pass / Fail Pass Pass Pass Pass

3.1 1065 Lab Audit 

The most important audits performed on the laboratory system were those directly related 

to the P M conc entration measurement ac curacy, na mely t he l inearity of f lows and the P M 

balance. The i ntake ai r flow and fuel flow were verified for l inearity, while t he C VS and PM 

sampling flows were verified using a p ropane recovery check. The linearity verifications were 

performed in accordance with 40 C FR Part 1065.307 a lthough they were performed within 180 

days r ather t han 370. T he pr opane c hecks w ere pe rformed w eekly dur ing of ficial t esting i n 

accordance with 40 CFR Part 1065.341. The maximum allowable interval for the propane check 

is 35 days. Table 8 shows the initial linearity verifications for fuel and intake air flow. 

One additional verification was performed for the fuel and intake air flow with similar 

results. Two different nominal flow rates were used for the PM secondary dilution system, one 

for steady state testing and another for transient testing. A total nominal flow of 3.6 m3/hr (2.0 

scfm) was used to target a filter face velocity of 100 cm/s during steady state testing. Because the 

filter measurement was unofficial the total flow was reduced during transient testing to prevent 

overloading the filter. A nominal total flow rate of 2.4 m 3/hr was used during transient testing. 

The propane recovery check was performed at a total secondary flow of either 3.6 or 2.4 m 3/hr 

depending on whether t he t esting at t he t ime w as s teady s tate or t ransient. Table 9 shows a 

summary of t he pr opane r ecovery results. The pe rcent di fference i s b etween the cal culated 

propane concentration based on a kno wn f low of pr opane a nd t he m easured pr opane 

concentration from a hydrocarbon analyzer. The pass limit for the CVS system is plus or minus 

two percent and plus or minus five percent for the secondary sampling system. 
REPORT 03.14936.12 33 of 174 

TABLE 9. CVS PROPANE RECOVERY CHECK SUMMARY

Test Date

CVS 

Blower, 

m3/hr

Secondary

Dilution, 

m3/hr

Secondary

Total, 

m3/hr

Secondary

Sample 

Diff, %

CVS 

Diff, %

10/29/08 3,941 1.75 3.53 -1.16 0.75

11/05/08 3,855 1.73 3.64 1.02 1.67

11/14/08 3,887 1.72 3.65 1.17 1.66

12/01/08 3,921 1.73 3.81 -1.1 0.37

12/08/08 3,931 1.72 3.75 0.06 1.33

12/18/08 3,955 1.75 3.62 -2.15 1.4

01/05/09 3,832 1.31 2.63 -1.89 -0.61

01/12/09 3,875 0.97 2.36 -2.72 -0.64

01/20/09 3,928 0.97 2.36 -2.98 -0.17

02/11/09 3,899 1.00 2.28 -0.05 -1.35

02/18/09 3,896 1.00 2.46 1.13 0.74

02/25/09 3,841 1.00 2.31 2.04 0.77

03/06/09 3,905 1.90 3.72 1.86 1.19

03/16/09 3,893 1.92 3.72 0.68 -0.56

03/24/09 3,804 1.05 2.17 0.6 0.51

04/07/09 3,927 1.87 3.87 1.37 0.28

Occasionally a p ropane check was out side of t he al lowable l imits, but when the ch eck 

was r epeated i t us ually pa ssed unde r t he s ame c onditions. It w as ne cessary t o pa ss t wo 

consecutive propane checks if the initial check failed. The final result from each day is shown in 

the Table 9.

Linearity w as a lso verified on the P M b alance us ed for f ilter w eights. T he line arity 

verification results are shown in Table 10. 

TABLE 10. LINEARITY VERIFICATION FOR PM BALANCE

PM Balance

Description |xmin(a1-1)+ao| a1 SEE R2

Measured 0.00 1.00 0.00 1.000

Criteria 20.00 0.99-1.01 20.00 0.990

Pass / Fail Pass Pass Pass Pass

Verification of t he P M ba lance i s r equired e very 370 da ys although the che ck was 

performed after 180 days for quality purposes and passed with similar results. 
REPORT 03.14936.12 34 of 174 

3.2 1065 PEMS Audit

Before the start of engine testing, each PEMS was required to pass the verifications set 

forth in CFR Part 1065. Because the measurement of PM is a non-standard process there are no 

audits s pecified on the actual P M me asurement, but the ins truments s till n eeded to pass t he 

necessary flow, temperature, and pressure audits. Table 11 lists the Part 1065 audits performed 

on the lab and on the PEMS. 

TABLE 11. SUMMARY OF PART 1065 AUDITS

Description CFR Reference Lab PEMS

Linearity 1065.307 x a x b

Torque Meter 1065.310 x

Pressure, Temperature, Dewpoint 1065.315 x

Fuel Flow 1065.320 x

Intake Flow 1065.325 x 

CVS Verification 1065.341 x

PM Balance Verification 1065.390 x 

a

 Linearity for lab performed on flow meters, torque meter, pressures, and temperatures 

b Linearity for PEMS performed on flow meters

Since no analyzer ve rifications w ere p erformed, the mos t c ritical a udits on the P EMS 

were the linearity verifications of flow measurements. Because all three types of PEMS use flow 

measurements to calculate a di lution ratio, the a ccuracy of the f low me asurements is di rectly 

related to the accuracy of the reported PM emissions. The AVL and Horiba PEMS measure their 

total and dilution flow to calculate a dilution ratio while the Sensors PEMS measures the dilution 

and sample flow. The following equation is used to calculate the dilution ratio: 

Dilution Ratio = 

Total Flow

Sample Flow = Total Flow

Total Flow − Dilution Flow

3.2.1 Horiba Flow Audits

The Horiba PEMS has four flow measurements in the system: dilution flow, DCS flow, 

make-up air flow, and total flow. During typical operation all flows but the dilution flow are held 

constant. The dilution flow rate is varied to sample proportionally from the raw exhaust based on 

changes in exhaust flow. The CFR requires linearity verifications on s ample, dilution, and total 

flow (whichever two of the three that are measured) [2]. The DCS and makeup flows affect the 

accuracy of the dilution ratio and the filter flow, however both of these flows are maintained at a 

nominal value of 2 lpm it was decided not to perform a linearity v erification in this situation. 

The steering committee elected to perform a s pot check on the DCS flow to ensure its accuracy 

and verify the filter flow which includes both the total and make-up air flow. The filter flow is 

equal to the difference between the total flow and make-up air flow. Because the filter flow was 

designed to operate at a constant flow of approximately 28 lpm it was not practical or logical to 

verify the flow measurement over 10 e ven points down to zero flow as recommended in CFR 

Part 1065.307. After discussing the issue with Horiba and the steering committee it was decided 
REPORT 03.14936.12 35 of 174 

to verify the filter flow over a range of plus and minus ten percent of its operating target, using 

eleven steps. The maximum range that the dilution flow could vary was approximately 21 to 31 

lpm; the dilution flow was verified over this range in eleven even steps. The flow audits were 

initially performed with a TSI flowmeter, but this was replaced with a bubble flowmeter when it 

was discovered that the accuracy of the TSI flowmeter degraded as the pressure during the flow 

measurement deviated from atmospheric. The results from the linearity verifications are shown 

in Table 12 for the Horiba PEMS. 

TABLE12. LINEARITY VERIFICATIONS FOR HORIBA PEMS 

Verification Description Intercept Slope SEE R2

Horiba1

Dilution Flow

Measured

Linearity Criteria

Pass/Fail

0.29

031 

Pass

0.98

0.98-1.02 

Pass

0.05

0.62 

Pass

1.000

0.99 

Pass

Filter Flow

Measured

Linearity Criteria

Pass/Fail

0.07

0.30 

Pass

0.97

0.98-1.02 

Fail

0.03

0.60 

Pass

1.000

0.99 

Pass

Horiba2

Dilution Flow

Measured

Linearity Criteria

Pass/Fail

0.08

0.31 

Pass

1.00

0.98-1.02 

Pass

0.3

0.61 

Pass

0.999

0.99 

Pass

Filter Flow

Measured

Linearity Criteria

Pass/Fail

0.18

0.30 

Pass

0.98

0.98-1.02 

Pass

0.03

0.60 

Pass

1.000

0.99 

Pass

Horiba3

Dilution Flow

Measured

Linearity Criteria

Pass/Fail

0.05

0.31 

Pass

1.02

0.98-1.02 

Pass

0.04

0.62 

Pass

1.000

0.99 

Pass

Filter Flow

Measured

Linearity Criteria

Pass/Fail

0.01

0.30 

Pass

0.99

0.98-1.02 

Pass

0.04

0.60 

Pass

0.999

0.99 

Pass

 Horiba-1 na rrowly m issed pa ssing linearity ve rification for tot al f low w ith a s lope of 

0.97. This check was repeated several times without passing. However, the error as a percent of 

point was better than 0.5 percent for all eleven points. This result is a problem with applying the 

linearity criteria f rom Part 1065 to a flow that is not verified over the range f rom z ero to full 

scale. Given the excellent agreement on a point-by-point basis, the steering committee elected to 

proceed without taking corrective action. 
REPORT 03.14936.12 36 of 174 

 The linearity verification results for the Sensors PEMS are shown in Table 13. 

TABLE 13. LINEARITY VERIFICATIONS FOR SENSORS PEMS

Verification Description Intercept Slope SEE R2

Sensors1

Sample Flow

Measured

Linearity Criteria

Pass/Fail

-0.01

0.02 

Pass

1.00

0.98-1.02 

Pass

0.00

0.04 

Pass

1.000

0.99 

Pass

Major Dilution Flow

Measured

Linearity Criteria

Pass/Fail

0.05

0.09 

Pass

0.98

0.98-1.02 

Pass

0.08

0.17 

Pass

0.999

0.99 

Pass

Minor Dilution Flow

Measured

Linearity Criteria

Pass/Fail

-0.04

0.05 

Pass

1.01

0.98-1.02 

Pass

0.01

0.09 

Pass

1.000

0.99 

Pass

Sensors2

Sample Flow

Measured

Linearity Criteria

Pass/Fail

0.01

0.01 

Pass

1.00

0.98-1.02 

Pass

0.01

0.03 

Pass

0.999

0.99 

Pass

Major Dilution Flow

Measured

Linearity Criteria

Pass/Fail

0.07

0.07 

Pass

1.01

0.98-1.02 

Pass

0.05

0.13 

Pass

0.999

0.99 

Pass

Minor Silution Flow

Measured

Linearity Criteria

Pass/Fail

0.02

0.04 

Pass

1.01

0.98-1.02 

Pass

0.06

0.08 

Pass

0.998

0.99 

Pass

Sensors3

Sample Flow

Measured

Linearity Criteria

Pass/Fail

0.01

0.01 

Pass

1.00

0.98-1.02 

Pass

0.01

0.02 

Pass

0.999

0.99 

Pass

Major Dilution Flow

Measured

Linearity Criteria

Pass/Fail

0.01

0.07 

Pass

1.00

0.98-1.02 

Pass

0.05

0.14 

Pass

1.000

0.99 

Pass

Minor Dilution Flow

Measured

Linearity Criteria

Pass/Fail

0.01

0.02 

Pass

0.96

0.98-1.02 

Fail

0.01

0.04 

Pass

1.000

0.99 

Pass

The Sensors PEMS had the capability of performing a self-audit using 1065 criteria. An 

external T SI f lowmeter was pr ovided a s pa rt of t he S ensors e quipment a nd i ts m easurements 

were recorded by the Sensors software to linearity verifications on the dilution and sample flows. 

The total dilution flow is calculated b y the addition of the major and minor dilution flows, so 

these two measurements are audited independently. The Sensors PEMS was able to pass all but 
REPORT 03.14936.12 37 of 174 

the m inor di lution f low a udit f or S ensors uni t 3. T his ve rification w as pe rformed r epeatedly 

without passing. Since the total dilution flow would still pass a linearity verification in this case, 

no further action was taken. In the case of both the Horiba and the Sensors, the absolute accuracy 

of the flows are important in determining the mass of PM emitted. The AVL s ystem is a real 

time particle sensor rather than a proportional batch sampler. For this type of instrument, only 

the di lution ratio a nd n ot t he a bsolute a ccuracy of t he t otal a nd di lution f lows a ffect t he 

measurement a ccuracy. For t his r eason t he di lution r atio w as a udited i nstead of t he t otal and 

dilution flow rates which are used to calculate the dilution ratio. It was also not possible to vary 

the total flow, which is held constant during normal operation. Although the dilution ratio was 

maintained at a constant of 5 throughout official testing, a six point check was performed ranging 

from 2 t o 6 in steps of 1. Table 14 shows the results for the linearity verifications for the AVL 

units. 

TABLE 14. LINEARITY VERIFICATIONS FOR AVL PEMS

Verification Description Intercept Slope SEE R2

AVL1

Flow Based Dilution Ratio

Measured

Linearity Criteria

Pass/Fail

0.01

0.07 

Pass

0.96

0.98-1.02 

Fail

0.01

0.15 

Pass

1.000

0.99 

Pass

AVL2

Flow Based Dilution Ratio

Measured

Linearity Criteria

Pass/Fail

0.03

0. 70 

Pass

0.97

0.98-1.02 

Fail

0.02

0.14 

Pass

1.000

0.99 

Pass

AVL3

Flow Based Dilution Ratio

Measured

Linearity Criteria

Pass/Fail

0.05

0.07 

Pass

1.02

0.98-1.02 

Pass

0.00

0.14 

Pass

1.000

0.99 

Pass

The MSS dilution ratio was initially verified using both flow measurements as well as 

CO2 measurements. Because the AVL PEMS had an internal CO2 measurement it would have 

been pos sible t o us e a C O2 span bot tle t o ve rify t he di lution r atio, unf ortunately i t w as not 

possible to ever introduce an undiluted CO2 sample to CO2 sensor to provide a span. When the 

sample is undiluted, the CO2 cell is bypassed so that it does not measure. The results shown in 

Table 15 w ere generated us ing T SI flowmeters t o m easure t he t otal and di lution f low a nd 

calculate t he di lution ratio in the s ame m anner as t he P EMS. AVL-1 and A VL-2 w ere bot h 

unable t o pa ss t he s lope c riteria but w ere w ithin t hree pe rcent of poi nt a cross t he s ix po int 

verification. The s teering c ommittee a greed to accept the di lution r atio accuracy tol erance of 

three percent.

3.2.2 Exhaust Flow

Official linearity verifications were not conducted on t he PEMS exhaust flow meters at 

SwRI. The steering committee decided that a calibration from the manufacturer was sufficient so 

long as the flow meter was within five percent of the lab during engine testing. The three flow 

meters tested from Horiba and the three from Sensors were all within found to be within five 
REPORT 03.14936.12 38 of 174 

percent of the lab measurement. An unofficial linearity verification was performed using the data 

from the steady state engine testing. The exhaust flow measurement was averaged during each 

state measurement sample and compared with the laboratory measured value over the same time 

period (50-270 seconds depending on the engine condition). There were between 64 and 84 data 

points per exhaust flow meter. The linearity plots are shown in Figure 13 for PEMS-1, Figure 14

for PEMS-2, and Figure 15 for PEMS-3. 

FIGURE 13. LINEARITY CHECK ON PEMS-1 EXHAUST FLOW DURING STEADYSTATE ENGINE TESTING

FIGURE 14. LINEARITY CHECK ON PEMS-2 EXHAUST FLOW DURING STEADYSTATE ENGINE TESTING

y = 0.977x - 16.188

R2 = 0.998

y = 0.995x - 15.427

R2 = 0.999

400

500

600

700

800

900

1000

1100

1200

400 500 600 700 800 900 1000 1100 1200

PEMS Exhaust Flow (m3/hr, 20°C)

Lab Exhaust Flow (m3/hr, 20°C)

Horiba-1 Sensors-1

y = 0.995x - 17.619

R2 = 1.000

y = 1.021x - 37.608

R2 = 0.997

400

500

600

700

800

900

1000

1100

1200

400 500 600 700 800 900 1000 1100 1200

PEMS Exhaust Flow (m3/hr, 20°C)

Lab Exhaust Flow (m3/hr, 20°C)

Horiba-2 Sensors-2
REPORT 03.14936.12 39 of 174 

FIGURE 15. LINEARITY CHECK ON PEMS-3 EXHAUST FLOW DURING 

STEADY-STATE ENGINE TESTING

Each of t he s ix ex haust f low m eters w as able to pa ss t he s tandard e rror, s lope, a nd 

correlation coefficient c riteria f or a r aw exhaust f low m easurement s pecified in CFR P art 

1065.307 however only one was able to pass the intercept criteria (Sensors-3). This is likely due 

in part to the fact that the measurements did not extend below 470 m3/hr making it more difficult 

to pa ss a n i ntercept c riteria t hat a ssumes e venly spaced d ata poi nts e xtended dow n t o z ero. 

Conducting a linearity verification on the exhaust flow measurement during engine testing was 

for informational purposes only. 

y = 0.986x - 13.106

R2 = 0.999

y = 0.987x + 3.218

R2 = 0.996

400

500

600

700

800

900

1000

1100

1200

400 500 600 700 800 900 1000 1100 1200

PEMS Exhaust Flow (m3/hr, 20°C)

Lab Exhaust Flow (m3/hr, 20°C)

Horiba-3 Sensors-3
REPORT 03.14936.12 40 of 174 

4.0 ENGINE DYNAMOMETER TESTING AND RESULTS

4.1 Testing Objective

The purpose of the engine dynamometer testing was to characterize the bias and precision 

errors of t he P EMS dur ing s teady-state and t ransient en gine ope ration. D uring s teady-state 

engine operation, the PEMS measurements were compared with that of the CVS filter-based PM 

measurement to characterize the bias in each of the three PEMS. The transient engine testing was 

used t o de termine t he p recision o f each PEMS b y quantifying the v ariability of the P EMS 

measurement over a series of repeated transient NTE events.

4.2 Experimental Setup

4.2.1 Engine and Sampling System 

Preliminary t esting was pe rformed using a 6. 4 liter lig ht he avy-duty di esel en gine 

provided by Navistar, however the engine used to generate all official steady-state and transient 

data was a 2007 V olvo MP7 provided b y Volvo Powertrain. T he test plan initially c alled for 

official dynamometer testing to be performed on two different engines, but funding constraints 

reduced official testing to a single engine. The Volvo MP7 had a displacement of 10.8 liters and 

was rated at 280 kilowatts (375 horsepower). The engine was equipped with a variable geometry 

turbocharger (VGT) and a water-cooled high pressure exhaust gas recirculation (EGR) loop. The 

engine intake system was connected to a test-cell water-cooled intercooler. The engine is shown 

in Figure 16. 

FIGURE 16. VOLVO MP7 INSTALLED IN A CVS TEST CELL
REPORT 03.14936.12 41 of 174 

The engine was also equipped with a close-coupled diesel oxidation catalyst (DOC) and a 

diesel particulate filter (DPF) combination. For the purpose of producing higher PM emission 

levels, a bypass was created around the aftertreatment system to allow an adjustable amount of 

exhaust f low a round t he D PF. A DOC w as a dded t o t he b ypass s o t hat a ll of t he P M i n t he 

exhaust w ould pa ss t hrough an ox idation c atalyst s imulating a s cenario of a cracked D PF. A 

picture of the bypass is shown in Figure 17. 

FIGURE 17. DPF BYPASS WITH DOC 

The or iginal s tock af tertreatment w as l ocated in t he m ain leg of t he ex haust, while a 

separate catalyst was procured for the bypass leg. The DOC was 76 millimeters (six inches) in 

diameter with a length of 152 millimeters (12 inches). Three butterfly valves were placed in the 

exhaust s ystem t o c ontrol t he a mount of e xhaust pa ssing t hrough e ach l eg. T he D PF w as 

regenerated via an exhaust fuel injection system. For all testing, the b ypass was open to some 

degree, how ever, the b ypass l eg w as cl osed when active regenerations w ere p erformed on the 

DPF. The DPF bypass went through multiple iterations until the proper PM level was achieved 

during steady-state testing. A PM level of 0.025 g /hp-hr was easily obtainable during transient 

cycles, however it w as extremely di fficult to obtain this s ame P M le vel dur ing s teady-state 

engine operation. Table 15 lists the five different configurations of the bypass that were tested.
REPORT 03.14936.12 42 of 174 

TABLE 15. LIST OF DPF BYPASS CONFIGURATIONS

Iteration

Pipe 

Diameter Inlet Probe

DOC 

Diameter Outlet Probe

1 3 No 3 1", Upstream 

2 3 No 3 1", Downstream

3 3 3", Upstream None 1", Downstream

4 3 3", Upstream 6 1", Downstream

5 (Final) 4 3", Upstream 6 3", Downstream 

In the final configuration (iteration 5), two butterfly valves in the main leg of the exhaust 

were completely closed with only a one or two millimeter gap between the valve and the exhaust 

pipe. This not only forced a majority of the exhaust through the bypass, but significantly raised 

the exhaust backpressure. Based on measurements upstream and downstream of the bypass with 

the AVL PEMS, it was estimated that well over 50 percent of the exhaust was routed through the 

bypass in the final configuration. A diagram of the bypass system is shown in Figure 18. 

FIGURE 18. SCHEMATIC OF ENGINE DYNAMOMETER EXPERIMENTAL SETUP

The schematic is not to scale so that some distances may app ear incorrectly. There are 

more t han 10 pipe diameters be tween t he m ixing poi nt of t he b ypass a nd t he f irst P EMS 

sampling pos ition f or t he S ensors S emtech P PMD. Additionally, there are approximately 10 

pipe diameters b etween each of t he P EMS s ampling l ocations s o that an y f low di sturbances 
REPORT 03.14936.12 43 of 174 

caused by a different pitot tube or sample probe should not affect the other PEMS. The portion of 

exhaust between the first and the last sampling location is insulated to minimize the cooling of 

the e xhaust a nd t hermophoretic de position of pa rticles. T he l ast s ampling pr obe i s t hat of t he 

Horiba OBS-TRPM, which is just upstream of the entrance into the CVS tunnel. The gaseous 

PEMS units (Sensors Semtech DS and Horiba OBS-2200) were used only for data acquisition 

and t rigger s ignals. N o P EMS g aseous e missions w ere r ecorded dur ing t his pr ogram. T he 

Semtech DS was used with both the Sensors PPMD and the AVL MSS, while the Horiba OBS2200 was used in conjunction with the Horiba OBS-TRPM.

The outlet of the aftertreatment was routed to a constant volume sampling (CVS) tunnel 

for emissions measurement. The CVS consists of a positive displacement pump and an upstream 

heat exchanger; it was maintained at a nominal flow rate of 3,740 m 3/hr (2,200 scfm) for t his 

testing. The di lution a ir i s e xtracted t hrough a f ilter pa ck f rom a t emperature a nd hum idity 

controlled area. The po rtion of t he C VS t unnel t hat i s e xposed t o t he test c ell e nvironment 

upstream of the exhaust is insulated to prevent heating of the dilution air from the test cell. The 

particulate matter samples were extracted from the CVS as shown in Figure 18. 

Intake air flow was measured by a laminar flow element (LFE) with a maximum flow of 

1,700 m3/hr (1,000 scfm). The LFE was oriented so that there were 10 diameters of straight pipe 

before the inlet and after the outlet to minimize flow disturbances. The fuel flow was measured 

by a M icro-Motion flow me ter. The addition of i ntake a ir f low a nd f uel f low w as us ed t o 

determine the exhaust flow using the equations from CFR Part 89. The exhaust flow was used to 

calculate t he exhaust P M conc entration from t he C VS f ilter as w ell as a che ck on t he P EMS 

exhaust f low m eters. A r aw C O2 analyzer w as us ed i n c ombination w ith t he e xhaust f low 

measurement t o calculate a r aw carbon balance f uel f low w hich was com pared with t he 

measured fuel flow and the dilute carbon balance fuel flow as a quality check.

The PM sampling system consisted of a 47mm teflon membrane filter (Whatman Teflo), 

a fine metal screen backing, and a plastic filter cartridge. The total flow is operated at a nominal 

flow of 60 s tandard l iters pe r m inute ( 2.1 s cfm) w hich r esults i n a filter f ace ve locity of 

approximately 100 cm/s. The standard temperature and pressure used for all flow rates in this 

report is 20 °C and 101.325 kPa as specified in CFR Part 1065. The nominal dilution flow is 30 

slpm (1.1 scfm) which resulted in a dilution ratio of 2 and a sample flow of 30 slpm (1.1 scfm). 

The system maintained a constant dilution ratio and achieved proportionality through sampling 

from the CVS. The s ystem was heated to 47°C and had an approximate residence time of 0. 8 

seconds from the inlet of the sample probe to the filter. A cyclone with a 2.5 micron cutpoint at 

17 slpm was positioned just downstream of the sample probe. The aforementioned PM sampling

system was the laboratory reference used to generate all reference PM data used in this program. 

4.2.2 Sensors PPMD

The PPMD was the PEMS unit installed closest to the outlet of the bypass, approximately 

15 diameters downstream. An experiment was conducted to ensure the flow was fully mixed at 

this location as will be discussed later. The PPMD was installed in the horizontal orientation as 

shown in Figure 19. 
REPORT 03.14936.12 44 of 174 

FIGURE 19. PPMD INSTALLED IN THE TEST CELL

The P PMD w as e quipped w ith a l ong s traight pipe c ontaining its e xhaust f low me ter 

(EFM) just upstream of the 90 degree elbow from which the sample is extracted. The dilution in 

the S ensors s ystem t akes pl ace i nside t he i nstrument. T he m ost c ommon i nstallation of t he 

PPMD is in the vertical position, although it requires only the rotation of the moisture traps to 

properly operate the PPMD in the horizontal orientation. The Sensors PEMS was equipped with 

two s tages of di lution know n a s M PS1 a nd MPS2. A lthough s ome pr eliminary t esting w as 

conducted using both stages of dilution, the steering committee decided to use only a single stage 

of di lution for all of ficial te sts. The P PMD is a pr oportional s ampling system tha t va ries its 

dilution ratio inversely with exhaust f low to maintain a mini mum di lution r atio of 6 a t t he 

maximum e xhaust f low r ate o f a n e ngine. T he P PMD me asures P M u sing a Q uartz C rystal 

Microbalance (QCM), which charges the particles using a corona needle, deposits the particle on 

a Quartz Crystal, and then measures the change in frequency of the crystal to determine the mass 

deposited. The PPMD is a batch sampling device meaning it does not report PM concentration in 

real-time but instead reports a s ingle mass value per event. Because each crystal requires a pr e 

and post frequency measurement to determine mass, a total of eight crystals are included to allow 

for continuous op eration b y switching crystals. Crystal s ampling b egins as soon as the engine 

enters the NTE zone and stops as soon as the engine exits the NTE zone. One of the of the eight 

crystals was used as a r eference crystal to adjust the measurements for changes in temperature 

and pressure. This left up to seven crystals available for measurement although it was common to 

have one or two crystals not working on any given test. The timing of the samples during testing 

was designed around having a minimum of five working crystals available for measurement. The 

PPMD was included in the measurement allowance program because inertial microbalances had 

already been approved for PEMS applications in 40 CFR Part 1065. 
REPORT 03.14936.12 45 of 174 

4.2.3 AVL MSS

The AVL Micro Soot Sensor (MSS, also known as the Photo Acoustic Soot Sensor or 

PASS) w as i nstalled dow nstream of t he P PMD i n t he m iddle of t he ve rtical por tion of t he 

exhaust pipe leading to the CVS tunnel. The MSS is connected to the tunnel via a 2 meter heated 

sample line which was maintained at 52°C. The mixing of dilution air takes place in the dilution 

box just upstream of the sample probe so that the dilute sample is transported through the sample 

line. The MSS is shown in Figure 20. 

FIGURE 20. THE AVL MSS

The MSS consists of two boxes shown in the above figure. The top box is the measuring 

unit w hich c ontains t he r esonance c hamber f or t he s oot m easurement. T he bot tom box is t he 

conditioning unit which contains the s ample and dilution pumps as w ell flow controllers. The 

dilution pump is optional as the MSS can also provide dilution air via an external input of 300 

kPa of compressed air. The steering committee requested that the MSS operate using its internal 

dilution pump, since this is the way it would operate during in use testing. The MSS measures 

soot by heating the elemental carbon using a laser. When the soot is heated it emits a sound wave 

that is detected by a microphone. The MSS can report soot concentration on a 1Hz basis and uses 

a constant dilution ratio, which was set at 5 for all official testing. Because the MSS measures 

only soot and not total PM, it was included in this program as a partial participant. If both the 

Sensors and Horiba units could not complete the measurement allowance program it was to be 

considered for in-use. The AVL system does not have its own gaseous PEMS; or data storage 

device; i nstead i t de pends on t he g aseous i nformation f rom t he S emtech D S a nd sends i ts 

concentration signal to the Sensors Semtech DS where the necessary information is recorded. For 

all testing in this program a s ingle S emtech D S w as us ed to record the s ignals from bot h the 
REPORT 03.14936.12 46 of 174 

Sensors S emtech P PMD a s w ell a s t he A VL M SS. T he pr obe f or t he Horiba O BS-TRPM i s 

located approximately 1.65 meters (5 feet) downstream of the MSS. The Horiba system contains 

two separate exhaust pieces that are each about 150 mm (6 inches long). The upstream portion is 

a tail pipe adapter (TPA) which is a pitot tube exhaust flow measurement. The downstream pipe 

contained the probe for the PM sampling. The TPA and sampling probe can be seen in Figure 21. 

FIGURE 21. THE PROBE AND TPA FOR THE HORIBA OBS-TRPM

4.2.4 Horiba TRPM 

The Horiba system is a proportional sampling device that varies its dilution ratio in the 

same manner as the PPMD. The dilution air is introduced just downstream of the probe before 

the heated sample line. The point of dilution can be seen in Figure 21 where the three stainless 

steel lines converge into the stainless steel cylinder. The OBS-TRPM uses a T SI EAD (referred 

to here as a DCS) real time particle instrument to measure the particle concentration on a second 

by second basis and collects PM on a gravimetric filter simultaneously. The filter weight gain is 

used to provide a calibration constant to the real time particle signal and apportion the PM mass 

appropriately. The DCS instrument measures continuously, but the filter is designed to sample 

during v alid N TE event ope ration from t he s ame di luted exhaust s tream. The f ilter s ampling 

begins after five seconds in the NTE zone and will continue for a minimum sample time of 30 

seconds e ven i f t he engine i s no l onger i n t he NTE z one. Because t he E PA’s P M s tandard i s 

based on gr avimetric filter analysis, the Horiba system was included in the program. The OBSTRPM is comprised of several different boxes including the heated enclosure (HE), the diffusion 

charge sensors (DCS), the electrical enclosure (EE), and the mechanical enclosure (ME). The HE 

contains the 47mm filter holder, and the DCS is the real time particle sensor. Dilution air was 
REPORT 03.14936.12 47 of 174 

provided using a com mercially av ailable oi l-less c ompressor. T he H oriba O BS-2200 ga seous 

PEMS was used to log the ECM J1939 broadcast, measure the exhaust flow, and provide an NTE 

trigger to the TRPM to start filter sampling. The combined TRPM-2200 system contains a total 

of 6 box es plus an external compressor. T he components of the Horiba system are depicted in 

Figure 22. 

FIGURE 22. THE HORIBA OBS-TRPM AND OBS-2200 

The TRPM used the same 47mm Whatman teflo filter, metal screen, and plastic cartridge 

as the CVS system. All weighing and conditioning of both the CVS and the TRPM filters was 

conducted i n t he S wRI f ilter w eighing r oom. The f ilter w eighing r oom is ma intained at a 

temperature of 22 ± 1°C with a dewpoint of 9.5 ± 1°C in accordance with CFR Part 1065.190. 

Filters were stabilized in the weighing environment for at least 1 hour prior to both the initial and 

final weights. Each filter was weight was determined by the average of three weights on a scale 

with a resolution of 0.1 μg. 

4.3 Bypass Mixing Verification 

The flow from the DPF bypass was reintroduced into the main exhaust stream using a 76 

mm (3 i nch) pr obe f acing dow nstream w ith a n or ifice ne ar t he t ip of the pr obe t o pr omote 

mixing. T esting w as c onducted t o e nsure t he e xhaust f low w as f ully mixed pr ior t o t he f irst 

sampling l ocation, w hich w as oc cupied b y t he Sensors P EMS. T wo pr obe or ientations w ere 

created at the spot where the Sensors sample was extracted. One of the probe orientations was 

parallel to the upstream exhaust elbow and one was perpendicular to the elbow. The orientation 

of the AVL sample probe is shown in Figure 23. 
REPORT 03.14936.12 48 of 174 

FIGURE 23. EXPERIMENTAL SETUP FOR MIXING VERIFICATION

At e ach of t he pr obe orientations t he M SS w as us ed t o m easure t he e xhaust s oot 

concentration using a variable length probe that could traverse the length of the 127 mm (5 inch) 

exhaust pipe. There were five sample locations for each orientation as shown in Figure 24. 

FIGURE 24. MIXING VERIFICATION SAMPLE LOCATIONS

Each of t he pos itions w ere one i nch apart, w ith pos ition 3,8 being a t t he c enter of t he 

exhaust pipe. For a measurement, the probe was started at location 1, moving down to 5 and then 

back t o 1. T he pr obe w as t hen m oved t o pos ition 6 m oving dow n t o p osition 10 a nd ba ck t o 

position 6. T he M SS m easurement w as r ecorded f or 80 s econds a t e ach pos ition, w ith t he 

average of the last 30 seconds used for comparison. The steady-state modes with the highest and 
REPORT 03.14936.12 49 of 174 

lowest e xhaust f low r ates w ere c hosen t o p erform t he m ixing ve rification, t o e nsure pr oper 

mixing over the entire range of test conditions. 

The mixing verification was performed several times with different bypass configurations 

with similar results in all cases. Only the results from the final mixing verification are presented 

since the other results from different configurations are not relevant to the data in this report. The 

data from the final verification is presented in Figure 25. 

FIGURE 25. MIXING VERIFICATION RESULTS

The da ta i s pr esented as t he r atio of t he average conc entration at ea ch location to the 

centerline concentration. The data was normalized in this way not only to remove some of the 

engine va riability but also s o t hat t he t wo exhaust f low c onditions w ith di fferent s oot 

concentrations could be compared on t he same plot. Each data point represents the average of 

two samples taken at each location. The highest deviation from the centerline concentration was 

four percent at location 10, low flow which was considered acceptable, especially given that no 

trends were observed in the data. A paired t-test using equal variance was also performed on the 

data relative to the centerline position for the parallel orientation using the high and low flow. 

Locations 6 t hrough 10 pa ssed t he t -test, e xcept f or l ocation 10 a t low f low, a lthough t he 

difference in the mean value between locations 10 a nd 7 was less than 4 percent. The steering 

committee was s atisfied w ith the mix ing r esults and directed SwRI to move f orward with the 

program without any further modifications for the bypass setup. 

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1 2 3 4 5 6 7 8 9 10

Concentration/Centerline Concentration

High Flow Low Flow
REPORT 03.14936.12 50 of 174 

4.4 PEMS Loss Corrections

Each P EMS m anufacturer was given t he o pportunity t o c orrect t heir f inal P M 

measurement t o account f or v arious pa rticle l osses e ncountered du ring the s ampling p rocess. 

Any l oss c orrection ha d t o be pr esented t o t he s teering committee f or a pproval be fore i t w as 

implemented in the program. Sensors and AVL both chose to implement loss corrections, while 

Horiba declined to apply a loss correction to their data. 

4.4.1 Sensors PPMD Loss Correction

Sensors c onducted w ork unde r a separate pr oject at S wRI t o experimentally as sess t he 

losses in the PPMD. David Booker presented the results of this work along with the proposed 

Sensors loss corrections at the meeting on D ecember 10th, 2008 a t SwRI. The final PPMD loss 

correction included thermophoretic, electrostatic, and CVS loss factors. The CVS loss correction 

factor was meant to estimate the particle losses in the CVS system, since this is the standard to 

which t he P PMD i s c ompared. A lthough t ypical l oss c orrection factors w ill i ncrease t he 

estimated PM concentration, the CVS correction factor actually decreased the PPMD estimated 

concentration. The S ensors s trategy was to use the th ermophoretic and electrostatic los s 

corrections to determine what the true PM concentration is and then reduce that number by the 

amount of PM mass they believe will be lost in the CVS system. They did not wish to merely 

adjust to the correct concentration since the CVS system to which their instrument was compared 

did not correct for losses. A total CVS loss of 15 pe rcent was estimated by Sensors based on 

general experience rather than specific data. The total loss correction was estimated to increase 

the PM concentration by 5 to 10 percent when including the electrostatic and thermophoretic loss 

factors. The proposed loss corrections were accepted by the steering committee and implemented 

in t he S ensors P PMD post pr ocessor. A ll S ensors da ta i n t his r eport i ncludes t hese c orrection 

factors unless otherwise stated.

4.4.2 AVL MSS Loss Correction

The proposed AVL loss correction was presented at the meeting on November 12th, 2008 

at S wRI. The los s c orrection implemented by A VL w as int ended to correct f or the 

thermophoretic losses in the system and is based off a paper by Stratmann et al [6]. The equation 

for the loss correction is shown below: 

M mss ( ) ( )( )( )dttTLtqtt soot ∫ ∆+= ex 1)( + ex

( ) 











+

< = 300150 otherwise 0 if 150 T 

)/b(T-a

TL

The m agnitude of t he correction i s t emperature de pendent a nd w as e stimated t o be 

approximately 10 percent in most cases. This correction was accepted by the steering committee 

and implemented in the AVL Concerto post processor. 

At the meeting at SwRI on December 10th, 2008, AVL stated that their loss correction 

was currently capped at a maximum loss of 25 percent regardless of the calculated value. AVL 
REPORT 03.14936.12 51 of 174 

requested approval to remove this limitation and allow the equations output to be the correction, 

regardless of its magnitude. This change was accepted by the steering committee. 

4.4.3 AVL MSS Total PM Correction 

The AVL Concerto post processor includes a function that estimates the total PM based 

on t he m easured s oot, e xhaust t emperature, a nd total h ydrocarbon c oncentration, a long w ith a 

number of a djustable i nput pa rameters i ncluding t he vol ume of t he catalyst, t he l ight o ff 

temperature of the catalyst and the sulfur content of the fuel. Although this calculation was not 

approved for use with the official measurement allowance data, a portion of the AVL data was 

processed to examine the results from the total PM model. All AVL data presented in this report 

refers to the soot concentration corrected for losses unless otherwise stated. 

4.5 Steady-State Testing Procedure

Originally the test plan called for two different engines at three different emission levels 

of P M: D PF out , 0.02 g/hp-hr l evel, a nd a 0.03 g /hp-hr le vel. The D PF out le vel is s imply 

whatever the emissions happen to be with no bypass which was well below the 2007 standard of 

0.01 g /hp-hr. T he 0.02 and 0.03 g /hp-hr l evels would be s et b y adjusting t he D PF b ypass t o 

produce t he corresponding b rake-specific P M n umber f rom t he C VS filter. D ue t o f unding 

limitations the testing was reduced to a single engine at a single emission level. Because 0.03 

g/hp-hr will be used for the first year of compliance testing and 0.02 g/hp-hr is used with the 

following years, i t w as i mportant t o i nvestigate t he pe rformance of t he P EMS c overing t hese 

levels. For this reason an average of the two threshold PM levels, 0.025 g/hp-hr, was used as the 

target.

The obj ective of t he s teady-state t esting w as t o evaluate t he bi as and precision of t he 

PEMS using 180 data points for each PEMS manufacturer. The 30 points consists of six steadystate modes of engine operation (6), 10 repeats (10), one emission level (1), one engine (1), and 

three different PEMS units (3), 6*10*1*1*3=180. A PM steady-state error surface, ( ) g ΔSS PM m mol , 

was de veloped for each P EMS m anufacturer s o that t here a re t hree steady-state PM e rror 

surfaces for use with calculation methods 1 and 2. For calculation method 3, the AVL MSS will 

have a unique ΔSS PM m based o n the c alculations for m ethod 3. A s m entioned previously, the 

Sensors and Horiba PEMS will only use methods 1 a nd 2. T o determine the most suitable six 

steady engine modes for steady-state testing screening tests were performed using the 80 points 

Cummins cycle and measuring the PM levels with the AVL MSS and the TSI Engine Exhaust 

Particle S izer ( EEPS). The E EPS pr imarily pr ovides s ize di stribution information, but ma ss 

concentration can be inferred using an assumed density. Since the AVL is the only PEMS that 

can report a real time mass concentration without further efforts such as filter weighing and post 

processing it was chosen to perform the screening work. The Cummins cycle steps through 80 

steady-state engine modes as a transient cycle. The engine is stepped through 10 different speeds 

and eight en gine l oads a t each of t he s elected speeds. The m inimum s peed of t he c ycle i s t he 

minimum NTE speed, and the minimum torque is 30 pe rcent of the torque at the given speed 

meaning that the cycle effectively maps the NTE zone. Figure 26 shows the speed and torque 

points of the cycle. 
REPORT 03.14936.12 52 of 174 

FIGURE 26. SPEED AND LOAD FOR THE 80 POINTS CYCLE

The engine remained at each of the points for 88 seconds with a one second transition 

between modes. Because the speed and load were only incrementally changed in between each 

mode, it was assumed that each point would stabilize relatively quickly. The fact that this cycle 

was us ed onl y as a m ethod of s creening a lso contributed t o t he de cision t o c hange m odes 

quickly. Even with such short modes, the total cycle length was 2 hour s. To allow for purging, 

and calibrating the gas analyzers, the cycle was divided into two 40 poi nts cycles each lasting 

one hour. The gas analyzers were necessary because this cycle was also used to tabulate ECM 

fuel rate errors for this error surface. 

The PM emissions were estimated by measuring each of the 80 modes with the EEPS and 

MSS, then choosing two of the modes to perform a filter measurement and compare the ratio of 

the filter measurement to the EEPS and MSS as an estimate of the CVS filter BSPM at each of 

the 80 m odes. Although this method is not highly accurate, it provided a way to quickly obtain 

rough estimates of the engine PM levels over a wide range of speed and torque conditions. Of the 

initial 80 points, only 12 were estimated to produce brake-specific PM of greater than 0.02 g/hphr. The final six modes were chosen with the intent of covering as much of the NTE z one as 

possible while still maintaining high BSPM levels, and a range of PM exhaust concentrations. 

Several iterations of adjusting the bypass, taking filter measurements, and narrowing down the 

number of points occurred before the steering committee approved the final six points for steadystate testing. Originally the DPF bypass was adjusted to produce the 0.025 g/hp-hr of PM based 

on a filter measurement during a short version of the NTE transient cycle. The PM emissions at 

steady-state were much lower at the same bypass setting. Since it was desirable to conduct the 

steady-state testing at the same PM levels as the transient testing, it was necessary to adjust the 

system to allow a much greater amount of the flow through the bypass. Six points were chosen 

0

20

40

60

80

100

120

0 20 40 60 80 100 120

Engine Torque (%)

Engine Speed (%)
REPORT 03.14936.12 53 of 174 

out of t he t welve t o p rovide a range of P M concentration, exhaust f low r ates, a nd engine 

operating conditions. The six points that were chosen are shown in Figure 27 along with the NTE 

zone. 

FIGURE 27. FINAL SIX STEADY-STATE MODES

Although t he M SS w as used t o s creen t he 80 p oints, t he a ctual c oncentration a t e ach 

position was verified with CVS filter measurements before selecting the points. The exhaust PM 

concentration w as calculated b y m ultiplying t he CVS P M c oncentration by t he C VS di lution 

ratio. T he C VS di lution r atio w as c alculated b y di viding t he a verage C VS flow r ate b y t he 

average exhaust flow rate. The CVS dilution ratio ranged from 3 t o 7.5 r esulting in an overall 

dilution ratio of 6 to 15 when including the secondary PM filter dilution. 

The steady-state testing was conduc ted as a modal t ransient c ycle with e ach m ode 

repeated twice for a total of 12 modes per cycle. 6 different cycles were created with the order of 

the m odes r andomized in each cycle. Table 16 lists t he s ample or der of t he m odes i n t he 6 

steady-state cycles.

0

500

1000

1500

2000

2500

500 1000 1500 2000 2500

Engine Torque (Nm)

Engine Speed (rpm)

Lug Curve NTE S Line NTE T-S Line NTE P-S Line SS Points

Mode 1

Mode 2

Mode 3

Mode 4

Mode 5 Mode 6
REPORT 03.14936.12 54 of 174 

TABLE 16. SAMPLE ORDER FOR STEADY-STATE CYCLE TESTING

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6

Sample 1 3 1 3 3 4 6 

Sample 2 6 4 1 4 2 1

Sample 3 5 3 6 1 1 4

Sample 4 2 6 2 2 5 3

Sample 5 4 5 5 6 3 5 

Sample 6 1 2 4 5 6 2

Sample 7 5 2 4 6 5 4

Sample 8 2 1 2 4 3 1

Sample 9 1 4 3 1 4 6 

Sample 10 3 5 1 2 6 3

Sample 11 6 6 5 5 2 5

Sample 12 4 3 6 3 1 2

The steady-state testing was conduc ted as a ramped modal c ycle w ith the eng ine 

remaining at each operating condition for three minutes before the start of sampling. An external 

trigger f rom t he l ab w as pr ovided t o e ach of t he P EMS a nd t he C VS f ilter s ystem s o t hat 

sampling w ould be gin simultaneously for a ll i nstruments. T he e ngine t hen r emained a t t he 

operating condition for 5 seconds after the end of sampling to ensure that no delay in the end of 

sampling by any of the PEMS caused part of the transition period to be captured as a s ample. 

The engine remained at the condition for five seconds after sampling had finished to ensure all 

systems had finished sampling before the operating condition changed. The order of the modes 

was randomized and each mode was repeated twice within a single cycle for a total of 12 da ta 

points for each cycle. Six different c ycles were created, which would create a total of 72 data 

points. Although the target was only 60 valid points for each set of PEMS, in practice several 

cycles h ad to be r epeated to collect enough v alid data. An example o f o ne of the steady-state

cycles is shown in Figure 28. 
REPORT 03.14936.12 55 of 174 

FIGURE 28. EXAMPLE OF STEADY-STATE CYCLE

To ensure a comparably accurate filter weight, a filter weight gain of 100 μg was targeted 

for each mode. By collecting this amount of material on each filter, the weighing variability and 

tunnel background contribution could be minimized. Collecting more material than this for each 

steady state sample would have caused problems with the Horiba and Sensors PEMS by limiting 

the amount of time they could operate before switching filters or cleaning crystals. To produce a 

similar f ilter weight ga in at s ix di fferent s teady-state m odes w ith different m ass r ates, the 

sampling time was adjusted for each mode to meet this target. The sample time ranged from 50 

seconds to 245 seconds. Since the sampling time for each mode was different, the total length of 

time spent at each mode was different as well. The sample time for each mode for each round of 

the PEMS is shown in Figure 29. 

0

500

1000

1500

2000

2500

0 500 1000 1500 2000 2500 3000 3500 4000

Torque (Nm), Engine Speed (rpm)

Time (sec)

Speed Torque
REPORT 03.14936.12 56 of 174 

FIGURE 29. STEADY-STATE SAMPLE TIMES

Figure 30 shows t he ave rage f ilter w eight gain for al l s ix modes f or each of t he t hree 

PEMS.

FIGURE 30. CVS FILTER WEIGHT GAIN FOR STEADY-STATE TESTING

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

PEMS 1 245 70 125 190 160 100

PEMS 2 240 50 100 175 90 60

PEMS 3 240 50 100 175 90 60

0

50

100

150

200

250

Sample Time (sec)

0

20

40

60

80

100

120

140

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

CVS Filter Weight Gain (μg)

PEMS 1 PEMS 2 PEMS 3
REPORT 03.14936.12 57 of 174 

Before presenting any of the PM concentration data it should be mentioned the choice of 

units i n t his w ork. T he t est pl an c alled f or t he PM c oncentration t o be c alculated i n t erms of 

μg/mol. Although it is an uncommon unit for describing mass concentration it was considered 

for f undamental t o us e m ol f or vol ume r ather t han m 3 i n w hich case a s tandard r eference 

condition must be defined. For reference, 1 m g/m3 of air at 20°C and 101.325 kPa is equal to 

24.055 μg/mol. In several cases the values in mg/m3 are provided in parenthesis for reference, 

but all official data was calculated and plotted using μg/mol.

The preliminary steady-state results from PEMS 1 were presented at the December 11th

Figure 31

, 

2008 meeting at SwRI. After reviewing the first set of steady-state data, the steering committee 

felt t hat t he c oncentrations f rom t he s ix steady-state points w ere not e ffectively c overing t he 

desired range. Five of the modes are clustered between 115 and 161 μg/mol (4.8 and 6.7 mg/m3) 

with the remaining mode at 325 μg/mol (13.5 mg/m3). At the recommendation of the steering 

committee the bypass setting was slightly adjusted for PEMS 2 and PEMS 3 in an attempt to fill 

in s ome of t he r egion be tween 161 and 325 μ g/mol. D ue t o a s hift in t he e ngine out P M 

emissions, i t w as pos sible t o i ncrease t he c oncentration f or m odes 2, 3, 5, a nd 6 w hile 

maintaining the same levels for modes 1 a nd 4. I n fact, the dampers were actually adjusted to 

flow l ess exhaust through the b ypass indicating t hat the engine out PM h ad not onl y changed 

relatively between operating conditions but increased overall. shows the median CVS 

filter PM concentration for each of the three sets of PEMS.

FIGURE 31. STEADY-STATE EXHAUST PM CONCENTRATION (μG/MOL)

0

50

100

150

200

250

300

350

400

450

500

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

PM Concentration, CVS Filter (μg/mol)

PEMS1 PEMS2 PEMS3
REPORT 03.14936.12 58 of 174 

The m ode w ith t he hi ghest c oncentration, m ode 2, i ncreased up t o ne arly 423 μg/mol

while the lowest concentration increased from 115 μg/mol on mode 5 to 132 μg/mol on mode 4. 

It i s i nteresting t o not e that m ode 5 and 6 i ncreased i n c oncentration b y 87 a nd 64 pe rcent, 

respectively while mode 4 increased by less than 2 percent. This increase in concentration came 

while opening the valve in the DPF leg of the exhaust, thereby increasing the flow of exhaust 

through the DPF. Modes 2 and 3 both shifted downwards between PEMS 2 and PEMS 3 without 

any c hange i n t he e xhaust va lve pos itions. Figure 32 shows t he br ake-specific P M va lues as 

measured by the lab reference for all three sets of PEMS.

FIGURE 32. STEADY-STATE BRAKE-SPECIFIC PM, CVS FILTER (MG/HP-HR)

The brake-specific PM ranged from 15.7 mg/hp-hr to 43.5 mg/hp-hr.

4.6 Data Yield During Steady-State Testing

The test plan called for a minimum of 10 valid data points for each mode for each PEMS, 

allowing for a minimum data set of 30 for each 5th and 95th

0

5

10

15

20

25

30

35

40

45

50

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

Brake-Specific PM, CVS Filter (mg/hp-hr)

PEMS1 PEMS2 PEMS3

percentile delta that was generated. In 

reality the target of 10 data points per PEMS per mode was not met in all cases due to additional 

points that were invalidated by post processing software that had been updated after the testing 

had been completed. While very few data points were removed during post processing for the 

Horiba a nd AVL P EMS, S ensors s upplied S wRI w ith s everal n ew po st pr ocessors after t he 

completion of te sting tha t inva lidated a significant por tion of t he S ensors da ta. S ince t he 

information on what criteria would invalidate the data was not available at the time of testing, it 

was not possible to know how many additional tests would be required to achieve the necessary 

number of data points. The Sensor’s post processor was revised to include some points that were 

deemed va lid d ata but h ad be en excluded b y t he pos t pr ocessor. T he f inal s et of da ta f or t he 

Sensors PEMS had between 28 and 34 poi nts per mode. Figure 33 s hows the number of valid 

data points for steady-stat testing.
REPORT 03.14936.12 59 of 174 

FIGURE 33. NUMBER OF VALID DATA POINTS FOR STEADY-STATE TESTING

The t otal num ber of pos sible da ta poi nts w as e ither 40 or 41 de pending on t he m ode. 

Samples were taken 41 times at each mode, but modes 2, 3, 4, and 6 each had one point excluded 

due to mishandling of the CVS filter. 

4.6.1 Data Yield During Steady-State Testing

Table 17 shows t he s teady-state d ata yield by each of t he P M-PEMS r elative to the 

possible data yield obtained by the CVS. A total of 29 steady-state cycles were conducted for all 

three PEMS with each of the six steady-state modes repeated twice in each cycle for a total of 12 

data poi nts pe r cycle. Twenty-one of t hese cycles w ere cons idered valid tests f rom t he 

perspective of the function of the cycle command, NTE external trigger, filter sampling, and at 

least one or m ore of t he P EMS c apturing v alid da ta. D uring t hese va lid t ests f our da ta poi nts 

were m issed due t o a m ishandling of t he C VS f ilter, but t he r est w ere considered v alid da ta 

points from the perspective of the lab measurements. Data from the PEMS was removed for a 

variety of reasons i ncluding pr oblems w ith t he da talogging, s ampling, e xhaust f low 

measurement, mechanical failures, and filter handling. Appendix E contains a complete list of 

the reasons data was excluded, but several notable problems will be discussed here. 

0

5

10

15

20

25

30

35

40

45

Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6

Number of Valid Data Points

Horiba Sensors AVL
REPORT 03.14936.12 60 of 174 

TABLE 17. DATA YIELD BY EACH PM-PEMS

Possible Horiba

Sensors - 

NewPP

Sensors – 

Revised PP AVL

Mode 1 41 37 22 34 40

Mode 2 41 36 25 29 33

Mode 3 41 36 25 32 33

Mode 4 41 35 24 28 35

Mode 5 41 37 17 31 37

Mode 6 41 37 23 30 34

Total 246 218 136 184 212

The Horiba s ystem requires an external s ource o f com pressed air capable of s upplying 

approximately 30 l pm at 400 kP a or higher. A commercially-available oil-less compressor was 

provided b y Horiba for use with the s ystem. Unfortunately this compressor had a tendency t o 

stop working on qui te a few occasions during testing. Three different compressors of the same 

model w ere pr ovided a nd e ach e xperienced t his problem. It w as be lieved t hat t he compressor 

was overheating and shutting off to protect itself, although changes in the test cell temperature 

did not seem to influence its performance. The compressor would begin to work again after 10 to 

15 m inutes presumably once i t ha d c ooled of f. I f t he c ompressor s topped w orking w hile t he 

Horiba system was in operation, the system lost all of its dilution air once the small air tank had 

been depleted. This resulted in an undiluted exhaust stream being sampled onto the filter which 

would quickly overload the filter at the PM concentration levels used in this work not to mention 

fail pr oportionality r equired of t he H oriba s ystem. If t he c ompressor s topped w orking at a ny 

point dur ing an of ficial t est, t he H oriba da ta f or t hat t est w as voi ded. Figure 34 shows a n 

example of a steady-state cycle where the compressor stopped working. 

A problem occurred with Sensors 1 i nvolving the auto zero function of its exhaust flow 

meter. Every hour the Sensors system would attempt to zero the exhaust flow meter by switching 

the pressure transducers to ambient for a period of time less than a minute. On the Sensors 1 the 

solenoid switching the pressure transducers from exhaust measurement to zero was not working 

properly causing t he z ero function t o oc cur w hile pr essures w ere be ing m easured f rom t he 

exhaust. Because t he steady-state cycle l asted longer t han on e hour , this w ould c ause a n 

erroneous exhaust flow measurement on the last two modes of each steady-state cycle. Figure 35

shows an example of the EFM zero problems during a test. 

The EFM auto zero function was disabled for the remaining tests on S ensors 1, a nd for 

Sensors 2 and 3. 
REPORT 03.14936.12 61 of 174 

FIGURE 34. HORIBA SAMPLE FLOW TO THE FILTER AND DILUTION FLOW 

WHILE COMPRESSOR STOPS

FIGURE 35. SENSORS EFM ZERO DURING STEADY-STATE CYCLE

0

200

400

600

800

1000

1200

1400

1600

0 1000 2000 3000 4000

Exhaust Flow (kg/hr)

Time (sec)

Sensors Lab

EFM Zero
REPORT 03.14936.12 62 of 174 

4.7 Accounting for CVS Variability During Steady-State Testing

The s teady-state da ta h as va riability due to the P EMS, CVS, a nd test a rticle. By 

computing a paired difference for each data point, the test article variability is removed from the 

data. This concentration delta still contains variability associated with the PEMS and the CVS. 

The data collected in this program does not allow for independent assessment of the PEMS and 

CVS variability, but by assuming a CVS variability it becomes possible to assign the remaining 

variability to the PEMS. 

The following procedure for estimating the PEMS and CVS variability was proposed by 

Bill M artin at th e D ecember 12 th 2008 meeting at S wRI and w as a ccepted by t he s teering 

committee.

1. Since test article variation is the same for each individual observation by the CVS (xPEMS,i) 

and PEMS (xCVS,i

( ) i PEMS,i CVS ,i −=∆ xx

), compute the paired differences, 

. 

These paired differences (i.e., delta values or concentration deltas) contain random variation 

from the CVS, random variation from the PEMS, and a mean offset between CVS and PEMS 

(bias error). 

2. Divide the entire set of delta values into j = 1 to M subsets based on the values of χCVSi. The 

data sets are not subdivided by engine operating mode or PEMS serial number, but only the 

level of the reference concentration.

3. For e ach s ubset, t here a re i = 1 t o N j values. T he m edian and the M AD ar e us ed as the 

descriptive statistics.

4. Calculate t he m edian delta va lue, ∆ ,50 j , t he m edian a bsolute de viation of t he ∆i values, 

MADj , and the estimate of the standard deviation of the CVS random error, CVSrandom j SD , . 

 Ni j j median i ,1 ,50 )( = ∆=∆

 Ni j

MADj median ij j ,1 ,50 )( =

∆−∆=

 ∑

∑ =

=

= ⋅

j

j

N

i

CVS i

j

N

i

CVS filter i

j

CVSrandom j x

N

L

N

g SD

1

,

1

,,

,

1

1

5μ

 ⋅ ∑

∑

= =

=

j

j

N

i

N CVS i

i

CVS filter i

CVSrandom j x

L

g SD

1 ,

1 ,,

,

5μ

 CVS j

CVS filter j

CVSrandom j x

L

g SD ,

,,

,

5 = ⋅ μ

where LCVS filter,, i is the PM s ample f ilter loa ding. 5 μg is the a ssumed CVS va riability a s 

proposed by the steering committee based on a nominal filter loading of 100 μg. 
REPORT 03.14936.12 63 of 174 

For each subset j, calculate a corrected delta for each ∆i value, in subset j, as follows: 

 

ij j

j

j

j CVSrandom j

ijij j

j CVSrandom j

MAD

MAD SD

MAD SD

50

2 ,50

2

,

2

,50

2

,

2

else

45495.0( ) )(

then 

if 45495.0 

′ ∆=∆

∆+ ⋅− ′ ⋅∆−∆=∆

⋅>

Note that this approach correctly passes through any significant offsets observed in the data. 

These offsets should be passed through even if they persist only for a subset of data, such as 

a given mode. 

5. The entire set of corrected delta values is then to be used to establish the error surface for the 

steady-state data. The 5th, 50th, and 95th delta values are used to establish the 1st, 50th, and 

99th percentile values which are the inputs to the Monte Carlo model. 

4.8 Steady-State Testing Results

4.8.1 Comparison between PEMS and Lab Delta PM 

All steady-state data presented has already been corrected for the steady-state variability 

as m entioned above. The s teady-state c oncentration de ltas f rom P EMS 1 f or H oriba, S ensors, 

and AVL, are shown in Figure 36, Figure 37, and Figure 38, respectively. 

Each point on the x-axis represents the median exhaust PM concentration from the CVS 

filter f or a s ingle m ode. T he y-axis r epresents t he 5 th, 50 th, a nd 95 th

Figure 39

percentile of t he de ltas, 

PEMS – Lab. As mentioned previously there was a clear gap in the data between 162 μg/mol and 

325 μg/mol which represents a large portion of the target concentration range for a 0.025 g/hp-hr 

level. , Figure 40, and Figure 41 show the deltas for PEMS 2 where three of the modes 

were between 208 and 267 μg/mol. 

 Horiba-1 and Horiba-2 performed similarly with mode 2 showing a significant negative 

bias, mode 4 showing a positive bias and the other four modes closer to zero. Sensors-2 showed 

a much greater negative bias than Sensors-1. Sensors-1 had a 50th percentile of between 0 and -

26 μg/mol for five of the six modes while Sensors-2 was between -32 and -104 μg/mol for the 

50th percentile for the same five modes. In addition, mode 2 had a 5th percentile of -340 μg/mol 

at a reference concentration of 442 μ g/mol. For the same mode on Sensors-1 the 5th percentile 

was -157 μg/mol at a reference concentration of 326 μg/mol. AVL-2 was lower than AVL-1 with 

50th percentiles be tween -34 μ g/mol a nd -70 μ g/mol. T he 50 th

Figure 42

percentiles f or A VL-1 were 

between 13 μ g/mol a nd -32 μ g/mol. No i ndication of t he changes i n performance for t hese 

PEMS was discovered through the recommended checks and audits. The PM deltas for Horiba-3, 

Sensors-3, and AVL-3 are shown in , Figure 43, and Figure 44, respectively
REPORT 03.14936.12 64 of 174 

FIGURE 36. HORIBA-1 STEADY-STATE PM CONCENTRATION DELTAS

FIGURE 37. SENSORS-1 STEADY-STATE PM CONCENTRATION DELTAS

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Horiba 50th % Horiba 95th % Horiba 5th %

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Sensors 50th % Sensors 95th % Sensors 5th %
REPORT 03.14936.12 65 of 174 

FIGURE 38. AVL PM CONCENTRATION DELTAS FOR STEADY-STATE TESTING 

ON PEMS 1

FIGURE 39. HORIBA-2 STEADY-STATE PM CONCENTRATION DELTAS

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

AVL 50th % AVL 95th % AVL 5th %

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Horiba 50th Horiba 95th Horiba 5th
REPORT 03.14936.12 66 of 174 

FIGURE 40. SENSORS-2 STEADY-STATE PM CONCENTRATION DELTAS

FIGURE 41. AVL-2 STEADY-STATE PM CONCENTRATION DELTAS

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 100 200 300 400

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Sensors 50th % Sensors 95th % Sensors 5th %

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

AVL 50th % AVL 95th % AVL 5th %
REPORT 03.14936.12 67 of 174 

FIGURE 42. HORIBA-3 STEADY-STATE PM CONCENTRATION DELTAS

FIGURE 43. SENSORS-3 STEADY-STATE PM CONCENTRATION DELTAS

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Horiba 50th Horiba 95th Horiba 5th

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

Sensors 50th % Sensors 95th % Sensors 5th %
REPORT 03.14936.12 68 of 174 

FIGURE 44. AVL-3 STEADY-STATE PM CONCENTRATION DELTAS

 Horiba-3 and Sensors-3 are within the same ranges of their first two instruments, while 

AVL-3 p roduced a n e xtremely high bi as a t t he hi ghest concentration ( mode 2 ). T he 50 th

percentile o f m ode 2 for A VL-3 w as 84 μ g/mol, w hile i t w as 13 μ g/mol a nd -62 μ g/mol f or 

AVL-1 and AVL-2, respectively. 

When t he da ta f rom all three P EMS i s c onsidered, t he range of concentrations f rom 5 

mg/m3 to 18 mg/m3 is covered, although there is still a majority of the data located between 5 

and 7 mg/m3 as a result of the data from PEMS 1. The individual deltas for the Horiba, Sensors, 

and AVL are shown in Figure 45, Figure 46, and Figure 47, respectively. Please note that each 

point represents a single measurement, not pooled data. 

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

0 50 100 150 200 250 300 350 400 450

PM Concentration Delta (μg/mol)

Lab Reference Concentration (μg/mol)

AVL 50th % AVL 95th % AVL 5th %
REPORT 03.14936.12 69 of 174 

FIGURE 45. STEADY-STATE CONCENTRATION DELTAS FOR HORIBA

FIGURE 46. STEADY-STATE CONCENTRATION DELTAS FOR SENSORS

-400

-300

-200

-100

0

100

200

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Lab PM Concentration Reference (μg/mol)

Horiba-1 Horiba-2 Horiba-3

-400

-300

-200

-100

0

100

200

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Lab PM Concentration Reference (μg/mol)

Sensors-1 Sensors-2 Sensors-3
REPORT 03.14936.12 70 of 174 

FIGURE 47. STEADY-STATE CONCENTRATION DELTAS FOR AVL

Because t he r eference concentration of s everal of t he m odes w as similar, the s teering 

committee decided to group the data by reference concentration rather than by operating mode. 

The da ta w as s plit up i nto g roups of a pproximately 20 da ta poi nts ba sed on r eference 

concentration for de veloping t he steady-state PM er ror s urface. The H oriba da ta w as grouped 

into 11 sets, the Sensors data into 9 sets, and the AVL data into 10 sets. 

Figure 48, Figure 49, and Figure 50 show PEMS PM concentration plotted against the 

PM conc entration determined by t he r eference l aboratory f ilter m ethod for s teady-state N TEs. 

These plots mainly show the qualitative PEMS to PEMS scatter relative to the filter method. 

4.8.2 Correlation between PEMS and Lab PM 

Figure 51 shows a linear regression between each set of PM-PEMS and the CVS filter 

method. G ood c orrelation w as obs erved be tween t he M SS a nd lab, w ith as coe fficient of 

determination (R2

) of 0.84 and a s lope of 0 .89. The s lope suggests t hat t he M SS P M 

concentration is 11 pe rcent lower than that determined by the lab. This trend is expected since 

the MSS is measuring soot and the lab reports total PM. The slope seemed to be high because the 

MSS PM concentration in the range between 15 and 20 mg/m3 was higher than that of the lab. 

The linear regression between the Horiba TRPM and the lab resulted in R2

-400

-300

-200

-100

0

100

200

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Lab PM Concentration Reference (μg/mol)

AVL-1 AVL-2 AVL-3

of 0.55 and a 

slope of 0.86, indicating some correlation. After further investigation, it was recognized that the 

weak correlation was due to Mode 4 (high speed, light load) of the SS testing, with concentration 

levels of about 5 mg/m3. By removing this mode from the data, a R2 of 0.85 was obtained and the 

slope w as moved from 0.86 t o 0.88. It i s l ikely t hat M ode 4, could ha ve r esulted i n 

overestimation of PM due to nanoparticle formation with the Horiba dilution system, although 

CVS t esting a t t his condition di d not s how a n anoparticle m ode. Based on t he s lope of t he 

correlation, the Horiba PM concentration was 14 percent lower than that reported by the lab. 
REPORT 03.14936.12 71 of 174 

The linear regression between the Sensors PPMD and the lab results in R2 of 0.34 and a 

slope of 0.64. The weak correlation was due to data scatter. Except in the narrow range between 

5 and 7 mg/m3, the Sensors PPMD showed underestimated PM. Based on the slope, the Sensors 

PPMD PM concentration was 36 percent lower than that reported by the lab. 

FIGURE 48. STEADY-STATE HORIBA PEMS PM CONCENTRATION VERSUS THE 

LABORATORY REFERENCE

FIGURE 49. STEADY-STATE SENSORS PEMS PM CONCENTRATION VERSUS THE 

LABORATORY REFERENCE

0

50

100

150

200

250

300

350

400

450

500

0 100 200 300 400 500

PEMS PM Concentration (μg/mol)

Lab PM Concentration Reference (μg/mol)

Horiba-1 Horiba-2 Horiba-3 Lab Reference

0

50

100

150

200

250

300

350

400

450

500

0 100 200 300 400 500

PEMS PM Concentration (μg/mol)

Lab PM Concentration Reference (μg/mol)

Sensors-1 Sensors-2 Sensors-3 Lab Reference
REPORT 03.14936.12 72 of 174 

FIGURE 50. STEADY-STATE AVL PEMS PM CONCENTRATION VERSUS THE 

LABORATORY REFERENCE

FIGURE 51. LINEAR REGRESSION CORRELATION BETWEEN PEMS AND LAB

0

50

100

150

200

250

300

350

400

450

500

0 100 200 300 400 500

PEMS PM Concentration (μg/mol)

Lab PM Concentration Reference (μg/mol)

AVL-1 AVL-2 AVL-3 Lab Reference
REPORT 03.14936.12 73 of 174 

4.8.3 Steady-State PM Error Surfaces 

Figure 52, Figure 53, a nd Figure 54, s how t he or iginal c oncentration deltas t hat w ere 

presented to the steering committee on April 2nd, 2009 at SwRI. The Sensors data includes some 

additional poi nts t hat were a dded l ater due t o c hanging t he e rror tolerances of t heir pos t 

processor. Each marker on the plot represents one of the j = 1 to M subsets of data divided up by 

concentration value. Each plot shows both the 5th and 95th percentiles which are b ased on t he 

actual da ta. T he e rror s urfaces f or t he Monte Carlo m odel w ere ba sed on t he 1 st and 99 th

percentiles w hich were ex trapolated from t he 5 th and 95 th percentiles as suming a nor mal 

distribution of the data. 

The steering committee elected to smooth the error surface by not including some of the 

data points that were within the envelope of surrounding points. A similar decision was made on 

some of the error surfaces included in the gaseous measurement allowance program. 

Figure 55, Figure 56, and Figure 57 show the final steady-state error surface of Horiba, 

Sensors, and AVL, respectively. The data presented in these plots has already been processed to 

remove th e C VS va riability a nd adjusted to the 1 st and 99 th percentile f orm the 5 th and 95 th

percentile. T he lines on t he plot represent the final error surfaces as app roved by the steering 

committee. The Horiba and AVL error surfaces were accepted by the steering committee during 

the conference call on June 29th, 2009. Some of the sensors data was reprocessed with different 

tolerances to increase the data yield, so the sensors error surface was not accepted by the steering 

committee until the July 14th, 2009 meeting in Indianapolis. 

FIGURE 52. STEADY-STATE CONCENTRATION DELTAS FOR HORIBA

-200

-150

-100

-50

0

50

100

0 100 200 300 400 500 600

PM Concentration Delta (μg/mol)

Lab Reference Median PM Concentration (μg/mol)

5th % 95th % 50th %
REPORT 03.14936.12 74 of 174 

FIGURE 53. STEADY-STATE CONCENTRATION DELTAS FOR SENSORS

FIGURE 54. STEADY-STATE CONCENTRATION DELTAS FOR AVL

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

0 100 200 300 400 500 600

PM Concentration Delta (μg/mol)

Lab Reference Median PM Concentration (μg/mol)

5th % 95th % 50th %

-150

-100

-50

0

50

100

150

0 100 200 300 400 500 600

PM Concentration Delta (μg/mol)

Lab Reference Median PM Concentration (μg/mol)

5th % 95th % 50th %
REPORT 03.14936.12 75 of 174 

FIGURE 55. FINAL STEADY-STATE PM ERROR SURFACE – HORIBA

FIGURE 56. FINAL STEADY-STATE PM ERROR SURFACE – SENSORS

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 100 200 300 400 500

Corrected PM Delta (μg/mol)

CVS Filter PM Concentration (μg/mol)

Delta 1st % Delta 50th % Delta 99th %

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 100 200 300 400 500

Corrected PM Delta (μg/mol)

CVS Filter PM Concentration (μg/mol)

Delta 1st % Delta 50th % Delta 99th %
REPORT 03.14936.12 76 of 174 

FIGURE 57. FINAL STEADY-STATE PM ERROR SURFACE – AVL

With a few exceptions the PEMS generally exhibited a low bias, with the majority of the 

50th percentile deltas falling below zero. The Horiba and Sensors both showed a degree of level 

dependence with the negative bias generally increasing for higher PM concentrations. At a lab 

reference concentration of 413 μg/mol the Horiba 50th percentile was -92 μg/mol and the Sensors 

50th percentile was -177 μg/mol at a lab reference concentration of 403 μ g/mol. The AVL 50th

percentile w as ge nerally l evel i ndependent and remained between -8 a nd -58 μ g/mol f or t he 

entire r ange o f con centrations t ested. While t he 1 st percentile f or t he A VL also remained 

relatively constant t hroughout t he c oncentration r ange, t he 99 th percentile j umped to a m uch 

higher va lue at hi gh co ncentration. The A VL 9 9th percentile j umped from 25 μ g/mol at 291 

μg/mol to 165 μg/mol at 402 μg/mol. The cluster of positive deltas that caused this large increase 

in the 99th percentile data is due solely to mode 2 with PEMS 2. The majority of the positive 

deltas observed for the Horiba were due to mode 4 which was a high speed light load condition.

Since it is understood that the DCS (EAD) real time particle sensor in the Horiba system is more 

sensitive t o smaller pa rticles i t w as as sumed that m ay have a l arger n umber of s ub-50 nm 

particles than the other 5 modes tested, causing more of the filter mass to be attributed to mode 4. 

4.9 Transient Engine Results

The t ransient eng ine t esting w as de signed to characterize t he pr ecision error of t he 

PEMS. A transient cycle consisting of 30 N TE events, 32 s econds each, was repeated multiple 

times and any differences in the PEMS measurements were attributed to PEMS variability under 

the assumption that the engine operation and PM emissions remained constant. No lab reference 

value was captured on an individual NTE event basis, because it is not possible to collect PM on 

a CVS filter for each NTE. Instead filter measurements were taken for each integrated transient 

-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 100 200 300 400 500

Corrected PM Delta (μg/mol)

CVS Filter PM Concentration (μg/mol)

Delta 1st % Delta 50th % Delta 99th %
REPORT 03.14936.12 77 of 174 

NTE cycle as a general indicator of the PM emissions of the engine during the entire cycle. The 

cycle used in this testing was developed using the cycle generator from the gaseous measurement 

allowance program. In the gaseous program the cycle generator was used to develop 20 uni que 

cycles with different orders of NTE events and different transitions between events. The steering 

committee elected to generate one single cycle for the current work that would be used for all 

transient testing. Additional time was added in between NTE events to ensure that the PPMD 

could sample throughout the cycle without missing any NTE events under the assumption of 7 

working c rystals ( 6 f or s ample a nd one r eference c rystal). In a ddition s everal s hort non -valid 

NTE events were added to the cycle to challenge the PEMS for measurements of non-valid NTE 

events.

The D PF b ypass w as a djusted t o pr oduce a n i ntegrated cycle BSPM of appr oximately 

0.03 g/hp-hr based on the CVS filter. Instead of readjusting the bypass for each set of PEMS, the 

exhaust valves were set to the same position each time. The NTE events were all approximately 

34 s econds i n dur ation. T he or iginal e vents f rom t he c ycle generator were each 32 seconds, 

however the Sensors PEMS would occasionally see these events as shorter than 30 seconds and 

exclude t hem. Many of t he N TE eve nts cont ained extreme acc elerations and decelerations 

stopping just short of the lower boundaries of the NTE window. Considerable time was spent to 

ensure that the J 1939 signal remained in the N TE window during events, although the engine 

performance shifted slightly and occasionally an event was invalidated. If any NTE event did 

not remain in the NTE windows for at least 30 s econds it was not included in the data for the 

transient er ror s urface. A t otal of 16 c ycles were r un f or P EMS, 17 f or P EMS2, a nd 18 f or 

PEMS3. Figure 58 shows repeat engine speed traces for the transient cycle with the first and last 

official cycle conduc ted w ith each PEMS. Figure 59 shows t he r epeat t orque t races f or t he 

transient cycle. The COV on cycle work was 0.7 percent over 51 cycles. 

FIGURE 58. REPEAT ENGINE SPEED TRACES FOR NTE TRANSIENT CYCLE

0

5

10

15

20

25

30

35

600

800

1000

1200

1400

1600

1800

2000

2200

0 500 1000 1500 2000

Engine Speed (rpm)

NTE Event Number

Time (sec)

PEMS1-1 PEMS1-16 PEMS2-1 PEMS2-17 PEMS3-1 PEMS3-18 Event Number
REPORT 03.14936.12 78 of 174 

FIGURE 59. REPEAT ENGINE TORQUE TRACES FOR NTE TRANSIENT CYCLE

The exhaust concentration of soot, as measured by the AVL PEMS is shown in Figure 

60. The m easured exhaust c oncentrations of s oot r anged from 5 t o 380 0 μ g/mol ( 0.2 t o 160 

mg/m3) during NTE operations, with concentrations of 5 to 14 μg/mol (0.2 to 0.6 mg/m3) outside 

of the NTE window. The widest dynamic range observed during an NTE event was for event 9 

where the initial spike in soot concentration was measured as high as 3800 μg/mol before falling 

down to 55 μg/mol near the end of the event. Figure 61 gives a closer look at events 20 through 

23 to show the variability of the real-time signal over the three sets of PEMS. It should be noted 

that the AVL PEMS is shown here only because it reports the measured soot concentration on a 

second by second basis without additional processing. 

Although the Horiba PEMS was used in this program as a batch sampler rather than a 

second by second instrument for official results, Figure 62 shows the second-by-second data of 

the H oriba P EMS a long w ith t he A VL. T here were m ajor di fferences on s ome of t he pe ak 

concentrations most notably NTE event 9, w here the measured AVL concentration was nearly 

four times higher than the measured Horiba concentration. With no reference, the purpose of this 

figure i s onl y t o s how t hat t here a re di fferences w ithout t rying t o qua ntify t he a ccuracy. T he 

Sensors P EMS pe rforms i ts m easurements as a ba tch sampler s o that t here w as no real-time 

exhaust concentration reported. 

0

5

10

15

20

25

30

35

-200

200

600

1000

1400

1800

2200

0 500 1000 1500 2000

Engine Torue (Nm)

NTE Event Number

Time (sec)

PEMS1-1 PEMS1-16 PEMS2-1 PEMS2-17 PEMS3-1 PEMS3-18 Event Number
REPORT 03.14936.12 79 of 174 

FIGURE 60. REPEAT AVL SOOT CONCENTRATION TRACES FOR NTE 

TRANSIENT CYCLE

FIGURE 61. AVL SOOT CONCENTRATION DURING NTE TRANSIENT CYCLES, 

EVENTS 20-23 

0

5

10

15

20

25

30

35

0

500

1000

1500

2000

2500

3000

3500

4000

0 500 1000 1500 2000

NTE Event Number

Exhaust Soot Concentration (μg/mol)

Time (sec)

PEMS3-18 PEMS3-1 PEMS2-17 PEMS2-1

18

20

22

24

26

28

30

0

100

200

300

400

500

600

700

800

900

1000

1440 1490 1540 1590 1640

NTE Event Number

Exhaust Soot Concentration (μg/mol)

Time (sec)

PEMS3-18 PEMS3-1 PEMS2-17 PEMS2-1
REPORT 03.14936.12 80 of 174 

FIGURE 62. COMPARISON OF AVL AND HORIBA REAL TIME SIGNALS DURING 

TRANSIENT CYCLE

To properly quantify the precision of the PEMS it is important that the reference value 

remain constant otherwise changes in the source PM emissions will be attributed to the PEMS as 

measurement va riability. In examining the C VS filter r esults, it w as c lear tha t the e ngine P M 

emissions va ried somewhat dur ing te sting. After r eviewing the tr ansient P EMS da ta a t the 

January 28 th, 2009 meeting a t S wRI, the s teering c ommittee r equested that the i dea of a 

correction t o t he P EMS data be a pplied t o a ccount f or c hanges i n t he e ngine pe rformance. A 

correction factor ba sed on the c ycle i ntegrated CVS B SPM w as pr esented to the com mittee 

during the April 2nd

CFi = ∑ CVSi

N

i=1

N ∙ CVSi

= 

CVSavg

CVSi

, 2009 meeting at SwRI and was accepted for use on all of the transient data. 

The correction factor was calculated as follows:

The correction factor is multiplicative and applied to the PEMS data in the following manner:

PEMScorrected,i,j = CFi ∙ PEMSmeasured,i,j

 The CVS BSPM along with the correction factor is shown in Figure 63. 

0

20

40

60

80

100

120

140

160

180

0 500 1000 1500 2000

Exhaust Concentration (mg/m3)

Time (sec)

AVL-Soot Horiba-PM
REPORT 03.14936.12 81 of 174 

FIGURE 63. CVS BSPM AND CORRECTION FACTOR FOR TRANSIENT CYCLE

The correction factor ranged from 0.78 to 1.24. There were two cycles in which the CVS 

filter measurement was void; in these cases the correction factor was set to one. The CVS brake 

specific PM ranged from 34.4 mg/hp-hr to 21.7 mg/hp-hr with an average of 27.2 mg/hp-hr. The 

COV was 11.4 percent when cycles for all three PEMS are included. The COV was 7.2, 6.0, and 

4.8 percent for cycles from PEMS1, 2, and 3, respectively. 

While processing some of the AVL data and comparing to the lab, it was discovered that 

the incorrect CVS BSPM value had been used for one of the cycles. The value of 31.9 mg/hp-hr 

from cycle 1956 had also been used for cycle 1965 instead of the correct value of 28.4 mg/hp-hr. 

The entire set of transient CVS data was scrutinized again and no a dditional errors were found. 

This erroneous correction factor was included in the final transient error surface included in the 

model. The transient error surface was found to shrink for all PEMS by between 1 and 2 percent 

with t he c orrection a pplied. G iven t he s mall c hange i n t he out come no a ction w as t aken t o 

correct this mistake. The transient error s urfaces pr esented i n this report are t hose us ed i n the 

Monte Carlo model. 

Figure 64, Figure 65, and Figure 66 show the concentration deltas for Horiba, Sensors, 

and AVL, respectively. In the transient testing there was no lab reference to create deltas. Rather 

the delta was calculated from the 50th percentile of the lab data as follows:

Deltai,j = CFi * mPMi,j – 50th

i 

Where mPMi,j is the average flow-weighted PM concentration for the jth repeat of the ith NTE 

event. 50th

i is the 50th percentile of mPM for the ith NTE event. CFi

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

15

20

25

30

35

40

45

50

55

0 10 20 30 40 50 60

Correction Factor

CVS Brake-Specific PM (mg/hp-hr)

Transient Cycle Number

CVS Filter BSPM Correction Factor

is the previously described 

correction factor to adjust for engine variability.
REPORT 03.14936.12 82 of 174 

FIGURE 64. HORIBA CONCENTRATION DELTAS FOR TRANSIENT ENGINE 

TESTING

FIGURE 65. SENSORS CONCENTRATION DELTAS FOR TRANSIENT ENGINE 

TESTING

-150

-100

-50

0

50

100

150

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Median Flow-Weighted PM Concentration (μg/mol)

5th-50th Percentile 95th-50th Percentile

-150

-100

-50

0

50

100

150

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Median Flow-Weighted PM Concentration (μg/mol)

5th-50th Percentile 95th-50th Percentile
REPORT 03.14936.12 83 of 174 

FIGURE 66. AVL CONCENTRATION DELTAS FOR TRANSIENT ENGINE TESTING

The 5 th and 95 th percentiles of t he H oriba a nd AVL P EMS w ere boun ded b y pl us a nd 

minus 50 μg/mol. T he Sensors P EMS ha d s lightly l arger de ltas e xtending j ust be yond 100 

μg/mol for the 95th percentile and -150 μg/mol for the 5th percentile. The bias from the data was 

removed by subtracting the 50th from the 5th and 95th percentiles.

The steady-state error surface was designed to quantify the accuracy of the PEMS while 

the t ransient e rror s urface w as de signed t o qua ntify t he pr ecision of t he P EMS. H owever, a 

portion of the PEMS precision error is inherently captured in the steady-state error surface. At 

the meeting on J uly 15th, 2009 i n Indianapolis four different approaches were presented to the 

steering c ommittee f or r emoving the steady-state contribution t o t he pr ecision e rror f rom t he 

transient er ror s urface. T he s teering com mittee el ected to proceed w ith a pproach 3; f or 

simplicity the other three approaches will not be described. 

In this approach, the data from all three PEMS from the same manufacturer were pooled 

together (ie Sensors1, Sensors2, and Sensors3). This resulted in a total of 18 steady-state data 

points and 90 transient data points. A median and a MAD value were calculated for each of the 

108 data points and used to calculate a MAD relative error, transient effect, root mean squared or 

MADre,tr,rms. The MADre,tr,rms is defined as follows:

-150

-100

-50

0

50

100

150

0 100 200 300 400 500

PM Concentration Delta (μg/mol)

Median Flow-Weighted PM Concentration (μg/mol)

5th-50th Percentile 95th-50th Percentile
REPORT 03.14936.12 84 of 174 

MADre,tr,rms = �1

N� � MADtr,iMediantr,i

�

N 2

i=1

− 1

N� � MADss,iMedianss,i

�

N 2

i=1

Where tr denotes transient and ss denotes steady-state. Using an rms value eliminates the need 

for estimating the steady-state variability at each of 90 t ransient data points. The error surface 

was defined as the 90 percent confidence interval around zero (or no bias) as: 

5th Percentile = −1.65 ∙ PM ∙ MADre,tr,rms

95th Percentile = 1.65 ∙ PM ∙ MADre,tr,rms

Effectively the error surface is a straight line with a slope of +/- 1.65·MADre,tr,rms 

Figure 67

and an 

intercept of z ero. T he t ransient e rror s urface i s shown i n for H oriba, Figure 68 for 

Sensors, and Figure 69 for AVL.

FIGURE 67. FINAL HORIBA TRANSIENT PM ERROR SURFACE 

-120

-80

-40

0

40

80

120

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile
REPORT 03.14936.12 85 of 174 

FIGURE 68. FINAL SENSORS TRANSIENT PM ERROR SURFACE

FIGURE 69. FINAL AVL TRANSIENT PM ERROR SURFACE

-120

-80

-40

0

40

80

120

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile

-120

-80

-40

0

40

80

120

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile
REPORT 03.14936.12 86 of 174 

The median flow weighted PM concentration on t he x-axis of each graph is extended to 

the highest measured concentration observed by any of the PEMS during transient testing. The 

slope of the transient PM error surface is 0.083 (1.41) for Horiba, 0.228 (3.27) for Sensors, and 

0.099 ( 1.66) f or A VL. The 1 st percentile f or t he P EMS ar e -34, -92, a nd -40 μg/mol a t th e 

maximum PM concentrations observed by each of the PEMS during steady-state testing (around 

400 μg/mol) for the Horiba, Sensors, and AVL systems. The 1st percentiles during steady-state

testing at t he s ame P M level w ere -177, -366, a nd -110 μ g/mol f or t he Horiba, S ensors, and 

AVL, respectively. Because the steady-state error surface was much larger than the transient it 

tended to dominate the model results as discussed later.

4.10 CE-CERT Mobile Lab Correlation

The t est pl an c alled for t he v alidation of t he model t o be pe rformed b y the m obile 

emissions l aboratory ( MEL) ope rated b y t he U niversity o f C alifornia a t R iverside Bourns 

College of E ngineering C enter f or E nvironmental R esearch and T echnology ( CE-CERT). T he 

mobile laboratory consisted of a full-flow CVS system inside the trailer of a Class A truck. The 

mobile lab was capable of measuring gaseous emissions and filter based PM. The mobile lab was 

arrived at SwRI on A pril 9th

Figure 70

, 2009 t o compare brake-specific PM emissions and ensure that the 

reference during in-use validation is similar to the reference used during laboratory testing. The 

CE-CERT MEL was parked behind the SwRI test cell and an exhaust transfer line was fabricated 

to allow the M EL to measure the full e ngine e xhaust in the s ame w ay a s the S wRI te st cell. 

Because the CVS measurement technique requires the full flow of engine exhaust for emissions 

measurement, the exhaust system was designed in such a way that the exhaust pipe could easily 

be switched between the SwRI CVS and the MEL CVS using the same length and geometry of 

exhaust pipe. shows the exhaust system used for the CE-CERT correlation.

FIGURE 70. EXHAUST CONFIGURATION FOR CE-CERT CORRELATION

To CE-CERT CVS Tunnel

To SwRI CVS Tunnel
REPORT 03.14936.12 87 of 174 

Prior t o t he s tart of t esting, t he M EL C VS a nd SwRI C VS bot h unde rwent a pr opane 

check to ensure the sampling systems were operating at the correct flows. Further details of the 

audits performed by CE-CERT on t he MEL can be found in the CE-CERT report (reference?). 

The MEL was set to the same CVS flow, secondary dilution ratio, and filter face velocity as the 

SwRI C VS t o m ake t he m easurements as c lose as pos sible. O ne ke y factor t hat c ould not b e 

controlled was the dilution air for the MEL CVS. The SwRI CVS was able to draw flow that is 

conditioned and maintained between 20 and 30oC. The MEL CVS drew its dilution air from the 

ambient w hich pr evented t he c ontrol of t emperature a nd r elative hum idity of t he di lution a ir. 

Each sampling system was conditioned by sampling for 10 hours during steady-state DPF engine 

operation a t a hi gh e xhaust t emperature. A ctive D PF r egeneration oc curred dur ing t his 

conditioning period. A total of 16 s hort NTE transient cycles were conducted using each CVS 

system. The short NTE transient cycle was a m odified version of the NTE transient cycle used 

for official transient PEMS testing. The short cycle included 16 of the original 30 N TE events 

and lasted 755 s econds compared to 2130 s econds for the full NTE cycle. Table 18 shows the 

order of testing. 

TABLE 18. TEST PROCEDURE FOR CE-CERT CORRELATION

Active

Regen Cycle 1 Cycle 2 Cycle 3 Cycle 4

Active

Regen Cycle 1 Cycle 2 Cycle 3 Cycle 4

Day 1 1 SwRI SwRI SwRI SwRI 1 CE-CERT CE-CERT CE-CERT CE-CERT 

Day 2 1 CE-CERT CE-CERT CE-CERT CE-CERT 1 SwRI SwRI SwRI SwRI

Day 3 1 SwRI SwRI SwRI SwRI 1 CE-CERT CE-CERT CE-CERT CE-CERT 

Day 4 1 CE-CERT CE-CERT CE-CERT CE-CERT 1 SwRI SwRI SwRI SwRI

A manually triggered active DPF regeneration was performed before the start of each set 

of four test cycles to maintain a similar PM loading level on the DPF. Each day the test order 

was switched so that the SwRI tests were first on days 1 and 3 and the CE-CERT tests were first 

on days 2 and 4. Each test was conducted as a hot-start with a 20 minute hot soak in between test 

cycles or in between the DPF regeneration and the first test cycle. Figure 71 shows the brakespecific PM results from the 16 cycles.

 The data from test 9 for CE-CERT was removed due to a filter weight that was deemed to 

be an outlier. The average SwRI BSPM was 0.0287 g/hp-hr with a COV of 5.2 percent based on 

16 repeats. The average CE-CERT BSPM was 0.0265 g/hp-hr with a COV of 3.5 percent based 

on 15 r epeats. T he r eported C E-CERT e missions w ere on average 7.7 p ercent l ower t han t he 

SwRI reported em issions. The average reported B SCO2 by C E-CERT was 2.6 p ercent l ower 

than average SwRI reported value. One possible source of discrepancy between the two systems 

was the heat loss in the exhaust pipe prior to its entrance into the CVS. The SwRI system was 

completely sheltered within the t est c ell, while the C E-CERT e xhaust pi pe w as pr otruding 

outside in such a way that it was exposed to wind and ambient temperature effects. This could 

have resulted in a higher thermophoretic deposition of particles inside the exhaust pipe resulting 

in lower emissions for the CE-CERT system. However, the test plan stated that agreement within 

ten percent was considered sufficient, so the correlation was considered complete and the issue 

was not investigated further. 

A series of tunnel blanks were measured from both systems over sample periods of 15, 

30, and 60 minutes. Figure 72 shows the filter weight gains as a function of sample time.
REPORT 03.14936.12 88 of 174 

FIGURE 71. BRAKE-SPECIFIC PM RESULTS FROM CE-CERT CORRELATION

FIGURE 72. CVS FILTER WEIGHT GAIN DURING TUNNEL BLANKS

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Brake-Specific PM (g/hp-hr)

SwRI CE-CERT

Day 1 Day 2 Day 3 Day 4

0

2

4

6

8

10

12

14

0 15 30 45 60 75

Filter Weight Gain (μg)

Sample Time (min)

SwRI CE-CERT
REPORT 03.14936.12 89 of 174 

The filter weight gains during tunnel blank operation were similar for the SwRI and CECERT tunnels. T he w eight ga ins r anged f rom 1 .9 t o 12.6 μg a nd g enerally de creased a s t he 

sampling time increased. The filter weight gains during the correlation testing were greater than 

300 μg making any differences in the tunnel blanks insignificant. 

4.11 Investigation of DPF Regeneration 

The steering committee requested that screening tests be performed on measurements of 

active DPF regeneration. Although the majority of the time active DPF regeneration is excluded 

from i n-use m easurement as t he r egulations ar e cur rently written there ar e cer tain situations 

where regeneration could be included in a v alid NTE event. In addition, if the Horiba s ystem 

were t o t rigger f ilter s ampling dur ing D PF r egeneration t his w ould s till be i ncluded i n t heir 

calibration factor. So even if all NTE events containing active DPF regeneration were considered 

invalid, the measurement accuracy of the Horiba system during valid NTE events could still be 

affected through the filter calibration. For this testing the engine was operated at a medium speed 

medium l oad c ondition a t steady-state and the DPF w as a llowed to accumulate P M unt il th e 

ECM aut omatically t riggered a r egeneration. The va lves i n the b ypass w ere cl osed forcing a 

large majority of the exhaust through the DPF although the bypass was not completely sealed. At 

the poi nt w hen t he E CM i ndicated i t w as pr eparing f or a n a ctive r egeneration t he P EMS a nd 

CVS f ilter w ere t riggered t o s ample. T he P EMS s ampled 40 s econds on, f ive s econds of f 

throughout t he r egeneration, w hile t he C VS f ilter s ampled c ontinuously. Table 19 lists the 

brake-specific PM results of the PEMS and the CVS.

TABLE 19. PM EMISSIONS RESULTS FROM ACTIVE DPF REGENERATION

CVS Filter Horiba Sensors AVL

No. of Samples 1 36 31 35

Avg. BSPM, mg/hp-hr 6.8 4.6 7.1 0.2

It i s l ikely t hat onl y a s mall por tion of t he e missions dur ing t he r egeneration e ven 

included elemental carbon as shown by the fact that the AVL PEMS measured only three percent 

of t he e missions m easured b y t he C VS f ilter. T he H oriba a nd S ensors PEMS bot h measured 

BSPM values on t he same order as the CVS filter although the event-by-event emissions were 

much different. The EEPS also sampled continuously f rom the CVS through a long residence 

time secondary dilution tunnel with a nominal secondary dilution ratio of 2. T he EEPS number 

concentration was converted to a mass concentration assuming spherical particles with a density 

of 1 g/cm3. A comparison of the event-by-event emissions of the three PEMS and the EEPS are 

shown in Figure 73. 
REPORT 03.14936.12 90 of 174 

FIGURE 73. BRAKE SPECIFIC PM EMISSIONS DURING ACTIVE REGENERATION

Although the average brake-specific PM value from the Sensors system was very close to 

the lab value, it is unlikely that the Sensors instrument was properly capturing the behavior of the 

regeneration. T his i s be cause i ts be havior w as i nsensitive t o t he a ctive regeneration r egion a s 

shown i n F igure 73. F urthermore, t he i ndividual br ake-specific e missions s eemed t o be hi gh, 

particularly during the first three events and the last two events, where no regeneration occurred. 

The event-by-event emissions from the Horiba system had similar trend to the mass determined 

using EEPS number-weighted size distribution measurement, assuming spherical particles with a 

density o f 1 g/cm3. Comparing t he Horiba m easurements t o t he C VS f ilter a nd t he E EPS 

indicate that it was more accurate than the other PEMS at measuring the regeneration emissions. 

However, one aspect of the Horiba measurement that was not captured in this experiment, and 

that is the change in sensitivity of the D CS particle ins trument to different s ize particles. The 

filter calibration constant for the regeneration event indicated that the DCS was approximately 

15 times more sensitive to the particles emitted during the active regeneration compared to the 

particles emitted during steady-state engine testing. This means that if a steady-state engine cycle 

was s ampled onto the s ame f ilter as t he D PF regeneration the s ystem would be m uch less 

accurate. The Horiba system would over predict the emissions during the regeneration and under 

predict t he e missions d uring nor mal e ngine op eration. T hese f indings w ere pr esented t o t he 

steering committee at the December 11th

0

100

200

300

400

500

600

0

2

4

6

8

10

12

14

16

18

0 20 40 60 80 100

DPF Outlet Temperature (°C)

Brake-Specific PM (mg/hp-hr)

Regeneration Time (%)

Horiba Sensors AVL EEPS DPF Out Temp

, 2008 meeting in San Antonio. Although this could be a 

major issue in the accuracy of the measurement, due to budget limitation, the steering committee

did not add D PF r egeneration t esting t o t he pr ogram given t hat t he r egulated aspect of 

regeneration during in-use testing was vague. No data from the DPF regeneration investigation 

was included in the model. The AVL MSS was insensitive to regeneration events as shown in 

Figure 73. T his s uggests that the majority of ma ss emitted during r egeneration is likely to be 

volatile materials that will not be detected by the MSS.
REPORT 03.14936.12 91 of 174 

4.12 Investigation of Storage and Release 

At the s ame time as the D PF regeneration investigation, the s teering committee a lso 

requested that screening tests be performed in the area of storage and release of nanoparticles in 

the aftertreatment system. It has commonly been observed that volatile material emitted by the 

engine at low temperature will deposit on the DPF only to be released quickly when the engine 

exhaust temperature climbs. Three different DPF loading conditions were tested: 

• Low idle: 650 rpm, 65 Nm 

• Medium idle: 1200 rpm, 135 Nm 

• High idle: 1800 rpm, 135 Nm. 

 Idling times of 20, 60, and 90 minutes were tested at each condition. Following the period 

of low temperature loading, the engine was immediately brought to a high temperature condition 

to promote the release of the stored particles. Each of the three idling conditions was tested with 

two high temperature conditions: 

• Peak torque: 1200 rpm, 2170 Nm 

• Near rated power: 1800 rpm, 1425 Nm 

These t ests, s hown i n F igure 74 were con ducted as a s creening exercise w ith 

measurements b y t he EEPS t o determine w hich combination produced t he l argest r elease of 

nanoparticles to use for the PEMS testing. However none of the tested conditions resulted in any 

significant p article e missions on a num ber or mass ba sis. T o e nsure t hat t he E EPS w as not 

missing something that might have been captured by one of the PEMS, two tests were run with 

the PEMS: low idle to peak torque and high idle to peak torque. Only the low idle test is shown 

here because t he r esults a re ve ry s imilar. T he e ngine w as a llowed t o i dle f or one hour be fore 

going to peak torque for ten minutes; this process was repeated three times consecutively for a 

cycle length of 3.5 hour s. The PEMS only sampled during the peak torque portion of the cycle 

with the same 40 seconds sample, five seconds off cycling used in the DPF regeneration study. 

A spike of just over 2.0E6 particles/cm3 was observed during the transition from idle to 

peak torque although this appears to be due to acceleration and a possible slight misalignment 

between the di lution ratio and EEPS m easurement r ather t han a release of pa rticles f rom t he 

aftertreatment. The spike was less than 1.0E6 for the second and third transitions to peak torque. 

Figure 75 shows the brake-specific PM emissions measured by the three PEMS during the peak 

torque portion of the storage and release cycle. All samples are taken during the three repeats of 

the t en m inutes a t pe ak t orque a nd no e missions f rom t he i dle por tion of t he c ycle a re 

represented.
REPORT 03.14936.12 92 of 174 

FIGURE 74. TOTAL EXHAUST NUMBER CONCENTRATION DURING STORAGE 

AND RELEASE

FIGURE 75. BRAKE-SPECIFIC PM EMISSIONS DURING STORAGE AND RELEASE 

INVESTIGATION

0.0E+00

5.0E+05

1.0E+06

1.5E+06

2.0E+06

2.5E+06

0

500

1000

1500

2000

2500

0 2000 4000 6000 8000 10000 12000

Exhaust Number Concentration (part/cm3)

Engine Speed (rpm), Torque (Nm)

Time (sec)

Speed RPM Torque N-m EEPS

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

10.0

12.0

0 10 20 30 40

Brake-Specific PM (mg/hp-hr)

Sample Number

Horiba Sensors AVL

1st Repeat 2nd Repeat 3rd Repeat
REPORT 03.14936.12 93 of 174 

It is unclear why the Sensors were reporting emissions much higher than the other two 

PEMS. A single CVS filter was sampled for the entire ten minutes at peak torque for each repeat 

for a total of three filter measurements. The BSPM from the CVS filter was only between 0.3 

and 0.4 m g/hp-hr for each of the three r epeats. This was lower than any of the PEMS and an 

order of magnitude lower than the Sensors PEMS. The three negative emissions were attributed 

to t he s ame c rystal which w as unus ually noi sy during t he t est. A com parison of t he av erage 

BSPM values for the cycle is shown in Table 20. 

TABLE 20. AVERAGE BRAKE-SPECIFIC EMISSIONS DURING STORAGE AND 

RELEASE CYCLE

CVS Filter Horiba Sensors AVL

No. of Samples 3 39 36 39

Avg. BSPM, mg/hp-hr 0.38 0.54 2.97 0.41

The effect of storage and release on the PEMS emissions was not cl ear because of the 

inability to generate a s ignificant release of nanoparticles, but it was evident that investigating 

this phenomenon using this experimental configuration was not worthwhile. These findings were 

presented at t he D ecember 11 th, 2008 m eeting i n S an A ntonio a nd t he s teering committee 

declined t o pur sue a ny f urther w ork i n t his area. N o da ta f rom t he s torage and release 

investigation was included in the model. 

4.13 Engine Manufacturers Torque and Fuel Error Surfaces

The OEM supplied torque error surface was up dated from the gaseous PEMS program 

and t he O EM s upplied BSFC e rror s urface was replaced with a f uel f low er ror s urface. Five 

different engine manufacturers supplied data from 61 different engines. In addition data was used 

from t he f our engines t ested i n t he A CES pr ogram f or a t otal of 2,099 da ta poi nts f rom 65 

engines. T he E CM t orque de ltas w ere nor malized b y t he m aximum E CM t orque. T he t orque 

deltas are shown in Figure 76. 

The e rrors as a p ercentage of t he m aximum E CM t orque a re r elatively constant 

throughout t he e ntire m easured r ange i ndicating t hat t he e rror a s a pe rcentage of poi nt w ould 

increase as t he abs olute t orque de creases. The much smaller da ta s et us ed for t he ga seous 

measurement allowance program showed constant errors as a percentage of point rather than a 

percentage of maximum. The plot of ECM fuel flow errors as a percentage of maximum ECM 

fuel flow is shown in Figure 77. 
REPORT 03.14936.12 94 of 174 

FIGURE 76. OEM SUPPLIED TORQUE ERRORS

FIGURE 77. OEM SUPPLIED FUEL FLOW ERRORS

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

0.0 0.2 0.4 0.6 0.8 1.0 1.2

(ECM Torque - Lab Torque)/Max ECM 

Torque

ECM Torque/Max ECM Torque

-30%

-20%

-10%

0%

10%

20%

30%

0.0 0.2 0.4 0.6 0.8 1.0 1.2

(ECM Fuel - Lab Fuel)/Max ECM Fuel

ECM Fuel/Max ECM Fuel
REPORT 03.14936.12 95 of 174 

The fuel flow errors were also constant in relation to max fuel flow rate above 20 percent 

fuel flow. Since the fuel flow is expected to remain well above 20 percent during NTE event 

operation, the smaller errors at low fuel flows were not considered important. Although the OEM 

supplied torque and fuel flow error surfaces were created as percentages of point for the gaseous 

measurement al lowance, t hese s urfaces w ere generated as pe rcentages of m aximum f or t he 

current program. The 1st, 50th, and 99th percentiles were calculated for torque and fuel flow and 

sampled in the model using a normal distribution. The error surface deltas are shown in Table 21. 

TABLE 21. OEM ERROR SURFACE DELTAS FOR TORQUE AND FUEL FLOW

Parameter

Percentiles

1st

% Point

, 50th

% Point

, 99th

% Point

,

Torque -7.6 -1.7 4.1

Fuel Flow -4.5 -0.1 4.9
REPORT 03.14936.12 96 of 174 

5.0 ENVIRONMENTAL TESTING AND RESULTS

A s eries of t ests w ere conduc ted to characterize t he P EMS r esponse unde r i n-use 

conditions such as changes in pressure, temperature, and humidity, as well as their response to 

electromagnetic and radio frequency interference and shock and vibration. 

Initially the te st pl an called for the E PA P M G enerator s ystem to be u sed as the P M 

source for all e nvironmental t esting. T he P M generator i s c apable o f p roducing s oot, vol atile 

hydrocarbons, and sulfuric acid to simulate the PM emitted from a diesel engine. However, after 

the system was operated at SwRI it was discovered that the PM generator was too tall to fit into 

the a ltitude c hamber. It was like ly th at it w ould not be pos sible to operate the P M generator 

under atmospheric conditions while the output was subject to a constantly varying pressure. The 

diffusion rate of the hydrocarbon vials is pressure dependant so it would not have been possible 

to m aintain a c onstant PM s ource w hile v arying t he pr essure i nside of t he chamber. S wRI 

proposed u sing a J ing mini-CAST s oot g enerator i n pl ace of t he P M g enerator. T he s oot 

generator is only a fraction of the size and much easier to operate compared to the PM generator. 

The steering committee agreed to allow the soot generator to be used for the altitude testing but 

requested that the PM generator be used for the temperature and humidity testing. 

It w as de cided to operate t he E lectromagnetic Interference / R adio Frequency 

Interference ( EMI/RFI) and s hock a nd vi bration t esting a s s creening. In t his c ase s creening 

testing me ant tha t the PEMS w ere ope rated while s ampling z ero air to look for pot ential 

problems. T he r esults f rom t he s creening t esting w ould t hen be pr esented t o t he s teering 

committee to decide whether to proceed with official testing to generate an error surface. The 

main motivation to conduct the EMI/RFI and vibration testing as screening was the result of the 

finding of the gaseous measurement allowance program that in most cases a failure mode of the 

PEMS was observed only as a malfunction of the system in which it could no longer operated. It 

was not commonly observed that the accuracy of the PEMS was affected while it continued to 

measure without detected problems. 

SwRI’s M echanical a nd M aterial E ngineering Division ( Division 18) performed the 

environmental testing. T he altitude, temperature and humidity testing was performed b y Rick 

Pitman and Mike N egrete. The E MI/RFI testing was p erformed by David Smith and Herbert 

Walker. T he s hock a nd vi bration t esting w as pe rformed b y D avid S mith, M ike N egrete, a nd 

Mark Orlowski. 

5.1 Reference Measurement Testing

An e ight hour b aseline measurement was p erformed f or comparison t o the e ight hour 

environmental tests. Unlike the gaseous measurement allowance program it was not possible to 

compare the accuracy of the PM-PEMS over the measurement period, only the variability. The 

PM and soot generators provide a particle source, but the correct concentration of the source was 

unknown. 
REPORT 03.14936.12 97 of 174 

During the eight hours of testing the PM concentration and dilution ratio was cycled to allow a 

more accurate assessment of the PEMS performance over a range of operating conditions. Three 

PM levels and four dilution ratios were sampled for a total of 12 test conditions. Figure 78 shows 

the schedule of target PM concentration and dilution ratio for one hour of environmental testing. 

FIGURE 78. TARGET DILUTION RATIO AND PM LEVEL PROFILE FOR 

ENVIRONMENTAL TESTING 

Each level was sampled at dilution ratios of 6, 12, 20, and 30 for the Horiba and Sensors 

PEMS. T he A VL P EMS w as m aintained a t i ts c onstant di lution r atio of 5. T he P EMS w ere 

maintained at a ta rget d ilution ratio for f ive mi nutes. With 35 seconds remaining, the s ample 

trigger w as ena bled for 30 seconds. T his a llowed f or f our m inutes a nd 25 s econds f or 

stabilization, 30 seconds for sampling, and five seconds after sampling to ensure sampling on all 

PEMS had stopped before the target dilution ratio was changed. The PEMS were cycled through 

the four dilution ratios at a single PM concentration level before the process was repeated at the 

next concentration level. It took one hour to cycle through each combination of dilution ratio and 

PM concentration. This profile was repeated eight times for a total test time of eight hours. The 

schematic showing the experimental setup can be seen in Figure 79. 

0

1

2

3

4

0

5

10

15

20

25

30

35

0 10 20 30 40 50 60

Target PM Level

Target Dilution Ratio

Time (min)

Target DR Target PM
REPORT 03.14936.12 98 of 174 

FIGURE 79. EXPERIMENTAL SETUP FOR ENVIRONMENTAL TESTING

The particles from the soot generator passed through a Krypton-85 neutralizer to bring 

the cha rge o f t he p articles t o a minimum B oltzmann distribution of c harge [ 7] to minimize 

particle losses due to electric forces. A large volume was placed downstream of the neutralizer 

to minimize the pressure fluctuations observed by the soot generator and also to smooth out any 

changes in concentration from the generator. One of the AVL PEMS units was placed outside of 

the environmental chamber upstream of the orifice and overflow so that it was isolated from the 

chamber conditions. This PEMS served as a reference to verify that the soot concentration from 

the ge nerator w as s table. T he A VL uni t w as chosen be cause i t c ould pr ovide a r eal t ime 

measurement of the soot concentration. This was not a guarantee that the total PM concentration 

from the generator was steady, but typically the volatile emissions from the generator would not 

fluctuate significantly without some change in the soot concentration. 

The mini-CAST soot generator was used as the particle source for the baseline testing. 

The num ber m ean di ameter w as approximately 70 nm ba sed on m easurement w ith t he E EPS. 

The three concentration levels were nominal concentration levels of 25, 75, and 125 μg/mol with 

approximately 30 percent organic carbon based on the OC/EC measurement. Figures 80, 81, and 

82 show the concentration measurements during the baseline testing for the Horiba, Sensors, and 

AVL PEMS, respectively. Figure 83 s hows a comparison between the reference M SS and the 

PEMS MSS which were measuring simultaneously during the baseline testing.
REPORT 03.14936.12 99 of 174 

FIGURE 80. HORIBA ENVIRONMENTAL BASELINE MEASUREMENTS

FIGURE 81. SENSORS ENVIRONMENTAL BASELINE MEASUREMENTS

-20

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Measured PM Concentration (μg/mol)

Observation Number

-20

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Measured PM Concentration (μg/mol)

Observation Number
REPORT 03.14936.12 100 of 174 

FIGURE 82. AVL ENVIRONMENTAL BASELINE MEASUREMENTS 

FIGURE 83. REFERENCE MSS ENVIRONMENTAL BASELINE MEASUREMENTS

-20

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Measured PM Concentration (μg/mol)

Observation Number

0

20

40

60

80

100

120

140

Measured PM Concentration (μg/mol)

0 20 40 60 80 100

Observation Number

Ref MSS PEMS MSS
REPORT 03.14936.12 101 of 174 

Although t he s oot c oncentration f rom t he generator w as l ikely s table based on t he 

measurements from both AVL PEMS it is not clear whether the total PM concentration remained 

constant throughout the test. In Figure 80, the Horiba PEMS exhibited a clear downwards trend 

in measured concentration as t he t est pr ogressed. The r eported PM conc entration f rom t he 

Sensors P EMS in F igure 81 appeared to be s omewhat i ndependent of t he act ual P M 

concentration sampled. The Sensors data was too scattered to either confirm or disprove the PM 

trend obs erved i n t he Horiba da ta. In a ddition t o t he general dow nwards t rend of t he 

concentration, t he H oriba da ta a ppeared t o s uggest t hat t he a ccuracy of the di lution r atio w as 

playing a r ole i n t he m easurement. In obs ervations 25 t hrough 96, t he r eported c oncentration 

decreased each time the dilution ratio target increased. This suggests that the Horiba PEMS is 

has either a positive error on lower dilution ratios or a negative error on higher dilution ratios. 

In Figure 83, it is clear that the PEMS and reference AVL units were both able to resolve 

the di fferences be tween t he t hree P M l evels cl early even showing s imilar r esponses t o small 

changes i n concentration. Figure 84 depicts t he r elationship between t he A VL r eference 

measurement and the AVL PEMS measurement.

FIGURE 84. REFERENCE AVL VERSUS PEMS AVL FOR ENVIRONMENTAL 

BASELINE

The correlation coefficient between the two measurements is excellent, better than 0.99. 

The slope of 0.78 is likely to be a combination of a difference in response of the two instruments 

and line losses between the two points of measurement. 

y = 0.7762x

R2 = 0.9991

0

20

40

60

80

100

120

0 20 40 60 80 100 120 140

AVL PEMS Concentration (μg/mol)

AVL Reference Concentration (μg/mol)
REPORT 03.14936.12 102 of 174 

5.2 Pressure Chamber Testing

The SwRI altitude chamber is capable of simulating altitudes up to 19.8 km. The chamber 

is 1.5 m in diameter and 2.1 m tall. Typically the simulated altitude chamber at SwRI is operated 

only unde r v acuum t o s imulate a ltitudes g reater t han t hat of S an A ntonio. T he g aseous 

measurement allowance test plan called for pressures up t o 101.87 kP a or 45 m eters below sea 

level. T he e levation of S an A ntonio i s a pproximately 240 m eters a bove s ea l evel w ith a 

barometric pressure near 99 (98.4 based on altitude) kPa. In the gaseous measurement allowance 

program, the altitude chamber underwent significant alterations to achieve positive pressures and 

still m any p roblems w ere en countered. The S wRI engineer i n cha rge o f t he altitude ch amber 

requested that only negative pressures be tested to preserve the integrity of their test equipment. 

Simply changing the positive pressures to ambient would have resulted in a large portion of the 

testing being conducted at nor mal a tmospheric pr essure. Instead a s light ne gative pr essure of 

94.3 kPa (610 m, 2,000 ft) was repeated twice, once at 1.8 hours and once at 7.2 hours. Figure 85

shows the original pr essure profile from the test plan as w ell as the modified profile that was 

used during testing. 

FIGURE 85. ORIGINAL AND REVISED PROFILE FOR ALTITUDE TESTING

Significant efforts were devoted to ensuring that a stable PM source could be generated 

that was insensitive to pressure. Because the soot generator contains an open flame operating at 

atmospheric pr essure, the pr operties o f t he flame, and hence t he p article generation, tended to 

change with the pressure of the outlet. By placing an orifice in the transfer line between the soot 

generator and the pr essure cha mber i t w as pos sible t o operate t he s oot g enerator at a hi gher 

pressure and maintain a constant pressure through adjustment of an overflow valve upstream of 

the or ifice. A djustments w ere onl y n ecessary du ring t he pr essure r amps to 82.7 kP a a nd 90.0 

kPa. The valve was adjusted to maintain a constant pressure upstream of the orifice. Figure 86

shows the setup outside of the altitude chamber. 

-800

-300

200

700

1200

1700

80

85

90

95

100

105

110

0 1 2 3 4 5 6 7 8

Altitude (m)

Pressure (kPa)

Time (hr)

Original Revised
REPORT 03.14936.12 103 of 174 

FIGURE 86. ALTITUDE TESTING CHAMBER

During the first practice run, the AVL 3 blew a fuse in its measurement unit. The test was 

stopped and the fuse was replaced, but i mmediately bl ew a gain indicating an electrical failure 

within the unit. The unit was replaced with AVL 2 and testing continued with the problem not 

observed again. Due to space constraints, the Semtech DS and Horiba OBS-2200 gaseous PEMS 

were located outside of the environmental chamber. Since the purpose of these devices was only 

communications, it was not considered necessary to test them inside the chamber. The external 

compressor us ed for t he H oriba di lution a ir w as a lso i nstalled out side the c hamber. A l arge 

compressor was supplied by SwRI to provide oil-less dilution air. The steering committee had 

requested t hat t he H oriba s upplied c ompressor s hould be us ed but then a greed t o allow t he 

replacement compressor after it was determined that the original compressor could not operate 

for the entire eight hours without shutting off. A picture of the PEMS installed in the altitude 

chamber is shown in Figure 87. 
REPORT 03.14936.12 104 of 174 

FIGURE 87. PEMS INSTALLED IN THE ALTITUDE CHAMBER

As mentioned previously, analysis of the environmental data was more difficult for the 

PM-PEMS pr ogram c ompared t o t he gaseous PEMS pr ogram du e t o t he l ack of a know n 

reference c oncentration. F or t his r eason, onl y the va riability o f t he P EMS m easurement i n 

comparison to its average was compared. The individual data points were plotted as well as the 

average levels to show how far each measurement deviated from the average. Figure 88 shows 

the concentration measurements by the Horiba.

FIGURE 88. HORIBA ENVIRONMENTAL PRESSURE MEASUREMENTS

80

82

84

86

88

90

92

94

96

98

100

0

20

40

60

80

100

120

140

0 20 40 60 80 100

Chamber Pressure (kPa)

PM Concentration (μg/mol)

Observation Number

Avg Horiba Horiba Pressure
REPORT 03.14936.12 105 of 174 

The Horiba concentration measurements were not consistent for different dilution ratios. 

In the baseline test it was observed that increasing the dilution ratio in the range of 6, 12, 20, and 

30 t ended t o de crease t he r eported c oncentration. In t he pr essure t est, it appe ars t hat t he 

measured concentration increases as the dilution ratio increases. The largest deviations from the 

average occurred during observations 53-60 while the pressure was at its lowest indicating that 

the a mbient pr essure doe s ha ve s ome ef fect on the accur acy of the H oriba r eported 

concentration. D uring t he pe riod b etween obs ervations 53 a nd 60, t he s ystem w as una ble t o 

maintain its target total flow rate of 30 slpm, as shown in Figure 89.

FIGURE 89. HORIBA TOTAL FLOW AND DILUTION RATIO DURING PRESSURE 

TESTING

The total flow dropped to approximately 28.7 slpm during this period which is below the 

acceptable t olerance a ccording t o t he m anufacturer, causing t he da ta t o be i nvalidated. T he 

system was however able to maintain proportionality during this part of the test. The short spikes 

in the total flow is from each switch from sample to bypass mode.

The S ensors da ta, a s s hown i n F igure 90, e xhibited a l arge a mount of v ariability w ith 

some da ta m ore t han a f actor of t wo hi gher a nd lower t han t he a verage. With t hat a mount of 

scatter it was difficult to visually discern an effect of pressure on the measurement.

The AVL data, as shown in Figure 91, was grouped tightly around the average except 

during t he l ow pr essure e xcursion a round observances 53 -60. Because of t he ex cellent 

repeatability of t he m easurement, the e ffect of pr essure on the m easurement w as r eadily 

apparent. As the ambient pressure decreased, the measurement decreased as well.

80

82

84

86

88

90

92

94

96

98

100

0

5

10

15

20

25

30

35

40

0 20 40 60 80 100

Pressure (kPa)

Dilution Ratio, Total Flow (slpm)

Observation Number

Total Flow Dilution Ratio Chamber Pressure
REPORT 03.14936.12 106 of 174 

FIGURE 90. SENSORS ENVIRONMENTAL PRESSURE MEASUREMENTS 

FIGURE 91. AVL ENVIRONMENTAL PRESSURE MEASUREMENTS 

80

82

84

86

88

90

92

94

96

98

100

0

50

100

150

200

250

0 20 40 60 80 100

Chamber Pressure (kPa)

PM Concentration (μg/mol)

Observation Number

Avg Sensors Sensors Pressure

80

82

84

86

88

90

92

94

96

98

100

0

10

20

30

40

50

60

70

80

0 20 40 60 80 100

Chamber Pressure (kPa)

PM Concentration (μg/mol)

Observation Number

Avg AVL AVL Pressure
REPORT 03.14936.12 107 of 174 

To determine w hether a n error s urface w ould be g enerated for ea ch PEMS, the M AD 

from the baseline was compared to the MAD of the pressure test. If the MAD of the pressure test 

was greater, an error s urface was generated to present t o the s teering committee. The pl ots of 

median versus MAD shown in Figures 92, 93, a nd 94 were presented to the steering committee 

at the meeting in Indianapolis on July 15th, 2009. 

FIGURE 92. AVL PRESSURE MEDIAN VERSUS MAD

FIGURE 93. HORIBA PRESSURE MEDIAN VERSUS MAD

y = 0.022x - 0.124

R2 = 0.999

y = 0.047x + 0.539

R2 = 0.413

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0 20 40 60 80 100 120

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

AVL-Baseline AVL-Pressure

y = 0.073x + 5.936

R2 = 0.427

y = 0.121x - 4.006

R2 = 0.991

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

0 20 40 60 80 100 120

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Horiba-Baseline Horiba-Pressure
REPORT 03.14936.12 108 of 174 

FIGURE 94. SENSORS PRESSURE MEDIAN VERSUS MAD

The M AD va lues were higher f or t he b aseline compared t o the p ressure t est f or bot h 

Horiba a nd S ensors i ndicating t hat no a dditional va riability in the me asurement c ould be 

attributed t o c hanges i n pr essure us ing t his t echnique. N o environmental e rror s urface w as 

calculated for either Horiba or Sensors. The AVL MAD values were higher for the pressure test 

compared t o t he ba seline, s o an e rror s urface w as de veloped. It s hould be not ed t hat t he 

variability of the AVL is much lower for the AVL compared to the Horiba and Sensors, however 

because of the high precision of the AVL measurements during the baseline it was still possible 

to discern the added variability due to changes in ambient pressure. The environmental pressure 

error surface was calculated using the same pooled rms technique that was used for the transient 

error surface. Figure 95 shows the error surface that was generated for environmental pressure 

on the AVL PM. The error surface was accepted for use by the steering committee during the 

July 15th

y = 0.155x + 15.834

R2 = 0.756

y = 0.111x + 15.755

R2 = 0.609

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

0 20 40 60 80 100 120 140

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Sensors-Baseline Sensors-Pressure

, 2009 meeting in Indianapolis. 
REPORT 03.14936.12 109 of 174 

FIGURE95. FINAL ERROR SURFACE FOR ENVIRONMENTAL PRESSURE AVL PM 

CONCENTRATION

This e rror s urface w as similar in magnitude to the A VL tr ansient P M e rror s urface, 

however it was much smaller than the AVL steady-state PM error surface.

5.3 Temperature and Humidity Chamber Testing

The temperature and humidity chamber testing was designed to characterize the effects of 

ambient temperature and humidity on the accuracy of PM measurement. The testing chamber has 

the capability of independently controlling the temperature and moisture content of the air in the 

range o f t ypically obs erved levels from ambient c onditions. T he t emperature a nd hum idity 

profile from the gaseous program was modified based on data acquired during CE-CERT testing. 

At the June 12th

-12

-8

-4

0

4

8

12

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile

, 2008 m eeting in Madison data was presented by Kent Johnson indicating that 

temperatures near the PEMS instruments could be above 60°C during in-use testing. With this in 

mind the steering committee elected to add temperatures as high as 50°C to the profile, where the 

original p rofile h ad a maximum t emperature of j ust ove r 30°C . T he or iginal a nd m odified 

temperature and humidity profiles are shown in Figure 96. 
REPORT 03.14936.12 110 of 174 

FIGURE 96. TEMPERATURE AND HUMIDITY PROFILE FOR ENVIRONMENTAL 

TESTING

The temperature chamber was unable to control the humidity under a temperature of 5°C 

so t he hum idity was un controlled dur ing t he po rtion of t he cycle w here t he t emperature w as 

between 2°C and 5°C. The moisture content was quite small during this portion of the cycle, so 

this was not considered to be a significant issue. Figure 97 shows the PEMS in the temperature 

and humidity chamber. 

FIGURE 97. PEMS INSTALLED IN THE TEMPERATURE AND HUMIDITY 

CHAMBER

0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

Relative Humidity (%)

Temperature (°C)

Time (hr)

Original Temp Modified Temp Original RH Modified RH
REPORT 03.14936.12 111 of 174 

The c ompressor pi ctured w as pr ovided b y S wRI f or us e dur ing environmental t esting. 

The or iginal c ompressor s upplied b y H oriba was m uch s maller, but a lso ha d di fficulty 

completing eight hours of operation without shutting off. The compressor was included in the 

temperature chamber so that the dilution air for the Horiba system would be affected by the same 

changes in temperature and humidity that the dilution air of the Sensors and AVL systems would 

experience. The P M s ample w as t ransported into the cha mber us ing a he ated sample l ine 

maintained a t 60°C . T he temperature w as s et s lightly above the ma ximum te mperature of the 

chamber so that a constant temperature in the sample could be maintained throughout the test. 

There w as a s mall por tion of t he e nd of t he t ransfer l ine t hat w as n ot he ated, but i t w as 

extensively i nsulated to minimize t emperature effects. The E PA P M ge nerator w as us ed in 

conjunction with the soot generator in this work to provide the particle source. The experimental 

setup was shown previously in Figure 79. The PM generator is shown in Figure 98. 

FIGURE 98. THE PM GENERATOR INSTALLED OUTSIDE THE TEMPERATURE 

AND HUMIDITY CHAMBER

The Horiba and Sensors gaseous PEMS were installed outside of the chamber along with 

the PM generator. The PM generator is designed to add volatile hydrocarbons, sulfuric acid, and 

water vapor to an elemental carbon source. An oxidation catalyst removed any volatile from the 

soot generator before volatile was added from the diffusion vial ovens. Unfortunately both the 

sulfur oven and the syringes injecting water malfunctioned during testing and neither was able to 

be quickly repaired. For the official temperature testing the particle source consisted of elemental 

carbon from the mini-CAST and volatile hydrocarbon from the PM generator. 

Figure 99 s hows t he i ndividual H oriba c oncentration m easurements along with a 

comparison of t he average concentration measurement f or e ach P M level. T he ch amber 

temperature is included for reference as well.
REPORT 03.14936.12 112 of 174 

FIGURE 99. HORIBA ENVIRONMENTAL TEMPERATURE MEASUREMENTS

The Horiba data plotted in Figure 99 showed a significant amount of variability although 

it is unc lear f rom thi s g raph whether the v ariability is di rectly r elated to the c hanging 

temperature. For ex ample, the m easurements be tween obs ervation 25 and 35 when the 

temperature is below 10 °C tend to be below the c ycle average. However, the second time the 

temperature dr ops be low 10°C a round obs ervation 70, t he m easurements a re a bove t he c ycle 

average. Figure 100 s hows t he c hanges i n f ilter di lution a ir a nd c hamber t emperature. T he 

behavior of these variables cannot clearly explain why the PM concentration behaved the way it 

did in Figure 99. 

FIGURE 100. HORIBA TEMPERATURES DURING ENVIRONMENTAL 

TEMPERATURE TESTING

0

10

20

30

40

50

60

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80 100

Chamber Temperature (°C)

PM Concentration (μg/mol)

Observation Number

Avg Horiba Horiba Temperature

0

10

20

30

40

50

60

0 20 40 60 80 100

Temperature (°C)

Observation

Filter Temp Dilution Air Temp Chamber Temp
REPORT 03.14936.12 113 of 174 

Figure 101 shows t he i ndividual S ensor c oncentration m easurements as w ell a s t he 

average concentration for each PM level. The temperature measurements for the Sensors PEMS 

are shown in Figure 101. The Sensors data displayed a high degree of variation from the mean 

with no c lear t rend r elated t o t emperature. N o valid da ta w as captured f or t he l ast hour o f 

operation because t he S ensors P EMS w as una ble t o maintain the cr ystals at a t emperature of 

50°C when the chamber temperature was above 47°C. 

FIGURE 101. SENSORS ENVIRONMENTAL TEMPERATURE MEASUREMENTS

The AVL data, shown in Figure 102, e xhibited excellent repeatability in comparison to 

the other two PEMS. Any temperature dependence by the AVL PEMS was extremely small. 

FIGURE 102. AVL ENVIRONMENTAL TEMPERATURE MEASUREMENTS

0

10

20

30

40

50

60

0

50

100

150

200

250

0 20 40 60 80 100

Chamber Temperature (°C)

PM Concentration (μg/mol)

Observation Number

Avg Sensors Sensors Temperature

0

10

20

30

40

50

60

0

20

40

60

80

100

120

140

160

0 20 40 60 80 100

Chamber Temperature (°C)

PM Concentration (μg/mol)

Observation Number

Avg AVL AVL Temperature
REPORT 03.14936.12 114 of 174 

The t emperature da ta w as com pared in the s ame m anner as t he pr essure da ta w ith the 

MAD values plotted in relation to the median, as shown in Figures 103, 104, and 105, for the 

Horiba, Sensors, and the AVL PEMS, respectively. The MAD of the Horiba baseline is slightly 

higher than during the temperature test meaning that an error surface was not generated in this 

case. The Sensors and AVL MAD was higher for the baseline at lower concentrations, but higher 

for t he t emperature data for hi gher concentrations. An attempt was m ade t o calculate an error 

surface f or S ensors, but t he pool ed r ms t echnique r esulted i n a s lightly hi gher va lue f or t he 

baseline indicating that an error surface was not necessary. The error surface calculated for AVL 

using the s ame technique is shown in Figure 10 6. The PM median concentrations tested w ere 

between 25 a nd 135 μ g/mol. I t w as ne cessary t o e xtend the e rror s urface out t o ne arly 500 

μg/mol to encompass the range of concentrations encountered during engine testing. The steering 

committee elected to cap the error surface at plus and minus 5.3 μg/mol because it was unclear 

whether t he errors w ould c ontinue t o i ncrease outside of t he c oncentrations obs erved i n t he 

temperature and humidity testing. Extending a straight line out to the median concentration of 

481 μg/mol would have resulted in a 5th percentile of 18.9 μg/mol. 

FIGURE 103. HORIBA TEMPERATURE MEDIAN VERSUS MAD

y = 0.073x + 5.936

R2 = 0.427

y = 0.084x + 3.106

R2 = 0.767

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

20.0

0 20 40 60 80 100 120 140 160

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Horiba-Baseline Horiba-Temperature
REPORT 03.14936.12 115 of 174 

FIGURE 104. SENSORS TEMPERATURE MEDIAN VERSUS MAD

FIGURE 105. HORIBA TEMPERATURE AND HUMIDITY MEDIAN VERSUS MAD

y = 0.155x + 15.834

R2 = 0.756

y = 0.040x + 22.144

R2 = 0.446

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

0 20 40 60 80 100 120 140 160

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

Sensors-Baseline Sensors-Temperature

y = 0.022x - 0.124

R2 = 0.999

y = 0.006x + 1.072

R2 = 0.962

0.0

0.5

1.0

1.5

2.0

2.5

0 20 40 60 80 100 120 140 160

MAD PM (μg/mol)

Median PM Concentration (μg/mol)

AVL-Baseline AVL-Temperature
REPORT 03.14936.12 116 of 174 

The AVL temperature data is shown in Figure 106 including the individual and average 

measurements as well as the chamber temperature.

FIGURE 106. FINAL ERROR SURFACE ENVIRONMENTAL TEMPERATURE AND 

HUMIDITY AVL PM

5.4 Electromagnetic and Radio Frequency Interference Screening

The electromagnetic inference (EMI) and radio frequency interference (RFI) testing was 

conducted a s a s creening e xercise w ithout t he us e of P M s ource. D uring t he gaseous 

measurement allowance program the majority of the problems encountered caused a malfunction 

of t he P EMS t o t he p oint w here i t w ould no l onger op erate. T he main pur pose of t he 

environmental testing was not to test the durability of the PEMS, but to quantify any errors that 

might occur during the operation of a PEMS that would cause a measurement error. The same 

series of t ests f rom t he g aseous E MI/RFI t esting w ere c onducted a s a s eries of i ndividual 

screening t ests us ing o nly HEPA f iltered air. Based on the s creening results i t r emained a 

possibility to conduct a full test cycle using a particle source to generate an error surface.

By providing filtered room air to the PEMS it w ould still has been possible to detect a 

wide range of possible measurement accuracy issues, although some problems may only present 

themselves when a particle sample is present. To shorten the test time, the PEMS were triggered 

continuously s o tha t ti me in between te sts c ould be mini mized. This me ant th at the Horiba 

system was continuously sampling on a filter, and the Sensors system was continuously sampling 

on a crystal. The Horiba system was able to sample continuously for eight hours or more on the 

filter s ince no particle s ource w as loa ding on the f ilter. However, t he Sensors s ystem ha d a 

maximum sample time that was adjusted so that it would cycle through the crystals every 120 

seconds. This was desirable since it exercised the operation of all eight crystals however some

issues surfaced due to this testing technique. The problems included: both high voltage power 

supplies turning on at the same time, two crystals sampling at the same time, no crystal sampling 

-12

-8

-4

0

4

8

12

0 100 200 300 400 500

PM Concentration Delta (

μg/mol)

Median Flow Weighted PM Concentration (μg/mol)

1st Percentile 99th Percentile
REPORT 03.14936.12 117 of 174 

even t hough on e w as a vailable. E ach of t hese pr oblems oc curred i nfrequently, a nd i t w as 

believed that the problems were a result of leaving the trigger on continuously for long periods of 

time s ince none of t hese i ssues were obs erved during en gine t esting. These were functionality 

issues that did not appear to have any impact on the accuracy of the measurement. 

The EMI and RFI testing, shown in Figure 107, was conducted in a radiation chamber, as 

shown in Figure 105, with walls covered with large cones of carbon impregnated foam designed 

to a bsorb r adiation a nd minimize r eflections. F our s tandard S ociety of Automotive E ngineers 

(SAE) te sts w ere c onducted: B ulk C urrent Injection, Radiated Immunity, E lectrostatic 

Discharge, and Conducted Transients. 

FIGURE 107. AVL PEMS IN THE RADIATION CHAMBER FOR EMI AND RFI 

TESTING

The S ensors a nd AVL PEMS w ere bot h de signed t o r un of f t he ve hicle’s 12 V olt 

electrical s ystem s o bot h s ystems w ere pow ered us ing a 12 V a utomotive b attery t hat w as 

continually charged using a 120 VAC charger supplied by Sensors. The Sensors PEMS was a 

12V system so it connected directly to the battery. The AVL system is powered by 120 Volts 60 

Hz AC power so a commercially available inverter was provided by AVL to convert the 12 volts 

DC into 120 volts AC. The Horiba system was designed to operate using a generator, therefore it 

was still powered using the 120 VAC wall outlets. 

5.4.1 Bulk Current Injection

SAE test J1113/4 Immunity to Radiated Electromagnetic Fields – Bulk Current Injection

was performed to determine the effect of electromagnetic radiation on the electrical cables of the 

PEMS. The specifications used are detailed in Region 2, Class B of the J1113/4 test protocol. A 

calibrated current probe was place around the electrical cable and used to inject RF current into 

the cable. Figure 108 shows a cable running through the bulk current injection probe. 
REPORT 03.14936.12 118 of 174 

FIGURE 108. BULK CURRENT INJECTION PROBE

The probe was positioned at 120 mm, 450 mm, and 750 mm from the cable connector to 

test the cable three times. For each test, the frequency of the current was stepped from 1 MHz to 

400 MHz using the following step sizes. 

 1 MHz to 10 MHz – 1 MHz step size 

 10 MHz to 200 MHz – 10 MHz step size

 200 MHz to 400 MHz – 20 MHz step size 

These were the maximum allowed step sizes according to the SAE protocol. As with the 

gaseous pr ogram a 5 s econd dwell t ime w as used to ensure t he electromagnetic f ield had 

stabilized before s witching to the ne xt f requency. T he pr obe w as c alibrated t o de liver 60 

milliamps of c urrent a s s pecified in the te st p rocedure. Figure 109 s hows t he bul k c urrent 

injection probe being used to test a cable on the Horiba PEMS. Figure 110 and Figure 111 show 

the sensors and AVL bulk current injection setup, respectively. 

FIGURE 109. HORIBA PEMS SETUP FOR BULK CURRENT INJECTION

Test Cable BCI Probe
REPORT 03.14936.12 119 of 174 

FIGURE 110. SENSORS SETUP DURING BULK CURRENT INJECTION TESTING

FIGURE 111. AVL SETUP DURING BULK CURRENT INJECTION TESTING
REPORT 03.14936.12 120 of 174 

The c ables tested on the H oriba s ystem w ere th e s ample line temperature, sample line 

heater power, he ated enclosure h eater power, sample trigger and exhaust flow signal, ethernet 

connection f rom t he D CS t o t he e lectrical e nclosure, e thernet c onnection f rom the T RPM t o 

2200, DCS 12 volt power, and two AC power cords. There were two main problems discovered 

when the bulk current injection was applied to the cables of the Horiba PEMS. When the line 

supplying power to the heater in the heated enclosure was probed, the reported total flow of the 

system and started t o r apidly f luctuate a round t he f requency o f 10 M Hz. T his f requency w as 

manually repeated after t he s weep and t he s ame be havior oc curred. Figure 112 shows t he 

response of several of the Horiba signals to the current injection. 

FIGURE 112. HORIBA FLOW DISTURBANCE FROM BULK CURRENT INJECTION

The noise in the system appeared to have originated with several of the temperature and 

pressure m easurements i ncluding t he f ilter f ace t emperature and total f low temperature s hown 

here. The total flow temperature and pressures began to fluctuate causing fluctuations in the total 

flow measurement, which in turn caused the dilution flow to fluctuate in an effort to maintain 

proportionality. T his pr oblem w as onl y obs erved on t he c able s upplying he ater pow er t o t he 

heated enclosure box.

The communications between the laptop and the PEMS was disrupted when the Ethernet 

cables w ere pr obed. S everal t imes t he c ommunications dr opped out c ompletely. T he p roblem 

was intermittent around 40 to 50 MHz and 140 to 160 MHz. 

The f inal m ajor pr oblem ex perienced by t he Horiba s ystem dur ing B CI w as a l arge 

amount of noise in the exhaust flow signal when the analog signal cable between the OBS-2200 

and OBS-TRPM was probed. The exhaust flow is measured by the Horiba gaseous PEMS OBS2200. The OBS-2200 outputs the exhaust flow measurement as an analog voltage which is then 

0

10

20

30

40

50

60

70

80

0

5

10

15

20

25

30

35

0 500 1000 1500 2000

Temperature (°C)

Flow (slpm)

Time (sec)

OB-PFSS_Total_1 [L/min] OB-PFSS_Dil_1 [L/min]

OB-PFSS_Filter_T [degC] OB-PFSS_Tot_T [degC]

10 MHz During Sweep 10 MHz Repeated
REPORT 03.14936.12 121 of 174 

read by the OBS-TRPM Over frequencies between 1 and 40 MHz the exhaust flow signal read as 

high as 3,000 kg/hr on the OBS-TRPM while the measured value on the OBS-2200 was less than 

10 kg/hr. The exhaust flow signal during the BCI sweep is shown in Figure 113. 

FIGURE 113. HORIBA EXHAUST FLOW NOISE ON ANALOG CABLE DURING 

BULK CURRENT INJECTION

It was clear that the problem involved only the transmission of the exhaust flow signal as 

an analog voltage between the two systems, not the actual measurement of the exhaust flow. For 

in use testing this problem would be easily detectable when the two exhaust flow data from the 

two different files are compared; however, the TRPM relies on the analog exhaust flow signal for 

adjusting i ts di lution r atio f or pr oportional s ampling. E rrors i n t he e xhaust f low s ignal w ould 

cause the TRPM to lose its proportionality, and therefore any data collected with the same filter 

sampling would be voided. 

The cables tested on the Sensors system were the communication cable from the PPMD 

to DS, and the DC power. As mentioned previously, many of the problems that were encountered 

with t he S ensors P EMS ha ppened i ntermittently a nd c ould not be reproduced a t any s pecific 

frequencies. It w as assumed t hat pr oblems i nvolving c rystal s ampling s witching, hi gh vol tage 

power s upplies, a nd f luctuations i n t he c orona current were not i nduced b y t he bul k c urrent 

injection. H owever, t he S ensors s ystem di d e xperience s ignificant c ommunication pr oblems 

when t he s erial c able be tween t he P PMD a nd DS uni ts w as pr obed. T he c ommunication w as 

repeatedly disrupted over the entire frequency range causing a loss of data. In nearly every case 

the c ommunication w as r ecovered onc e t he r adiation s ubsided a llowing nor mal ope ration t o 

continue. 

-500

0

500

1000

1500

2000

2500

3000

3500

0 200 400 600 800 1000

Exhaust Flow (kg/hr)

Time (sec)

OBS-TRPM OBS-2200
REPORT 03.14936.12 122 of 174 

The cabl es t ested on the A VL s ystem w ere t he com munications cabl e be tween the 

measuring and conditioning unit, the analog output to the Semtech DS, and two AC power cords. 

The BCI induced a significant amount of noise on the analog signal cable between the AVL MSS 

and the Sensors Semtech DS. The AVL system can log its own data, but the data used officially 

in this program and during in use testing is the data logged by the Semtech DS. Two signals are 

carried on t he analog output wire: the measured concentration and the dilution ratio. These two 

numbers are multiplied together for the final reported concentration. The measured concentration 

did not see significant noise, but the dilution ratio signal was strongly influenced by radiation in 

the range of 1-3 MHz and 40 M Hz. A comparison of the MSS measured dilution ratio and the 

dilution ratio recorded by the Semtech DS is shown in Figure 114. 

FIGURE 114. BCI NOISE ON AVL ANALOG OUTPUT CABLE

The dilution ratio doubled from 5 to 10 during current injection at 1-3 MHz. The data in 

the pl ot w as recorded d uring t he f requency s weep, a nd t he pr oblem w as r eplicated m anually 

afterwards. S ince t he t rue di lution r atio w as s till r ecorded i n t he M SS l og f ile, t he c orrect 

reported concentration could be recovered after testing.

5.4.2 Radiated Immunity

The r esponse of t he P EMS t o c ontinuous na rrowband e lectromagnetic f ields w as 

measured using SAE test J1113/21 Electromagnetic Compatibility Measurement Procedure for 

Vehicle Components – Part 21: Immunity to Electromagnetic Fields, 10 kHz to 18 GHz, 

Absorber-Lined Chamber. The ex act s pecifications of t he t ests pe rformed were dr awn from 

Region 2, C lass B of t he J 1113/21 pr otocol. Several di fferent a ntennae were us ed t o generate

electromagnetic r adiation over t he f requency r ange of 10 kHz t o 1 GHz. T he e lectromagnetic 

0

2

4

6

8

10

12

0 100 200 300 400 500

Dilution Ratio

Time (sec)

Semtech DS Analog DR AVL MSS DR

1-3 MHz

40 MHz
REPORT 03.14936.12 123 of 174 

susceptibility experts at SwRI recommended ending the test at a frequency of 1 GHz rather than 

18 GHz due to the very small probability of detecting any susceptibility above 1 GHz. This same 

approach was used in the gaseous PEMS program. The carbon impregnated foam walls of the 

radiation chamber was designed to absorb any radiation so that the PEMS would only see the 

direct r adiation ge nerated b y t he a ntenna. T he f ollowing s tep s izes were us ed dur ing t he 

frequency sweeps:

• 10 kHz to 100 kHz – 10 kHz step size 

• 100 kHz to 1 MHz – 100 kHz step size 

• 1 MHz to 10 MHz – 1 MHz step size 

• 10 MHz to 200 MHz – 2 MHz step size

• 200 MHz to 1 GHz – 20 MHz step size 

The S AE s tandard f ield i ntensity o f 50 vol ts/meter was us ed w ith bot h ve rtical a nd 

horizontal electromagnetic radiation orientations. Figure 115, Figure 116, and Figure 117 show 

the Horiba, Sensors, and AVL radiated immunity testing setup, respectively. 

FIGURE 115. HORIBA PEMS SETUP DURING RADIATED IMMUNITY TESTING
REPORT 03.14936.12 124 of 174 

FIGURE 116. SENSORS PEMS SETUP DURING RADIATED IMMUNITY TESTING

FIGURE 117. AVL PEMS SETUP DURING RADIATED IMMUNITY TESTING
REPORT 03.14936.12 125 of 174 

The H oriba P EMS e xperienced several pr oblems r elated to the r adiated immunity t est 

including a l oss of c ommunications between t he l aptop a nd t he i nstrument. T his pr oblem 

occurred at several different frequencies for both horizontal and vertical polarizations. Because 

the data is logged on t he laptop any loss of communication is also a loss of data. This would 

result in a voided test since the filter calibration relies on collecting data the entire time the filter 

is s ampling. A p roblem was a lso obs erved with the c ontrol of t he di lution r atio, a s s hown i n 

Figure 118. During EMI and RFI testing the Horiba dilution ratio was maintained constant at 6. 

FIGURE 118. HORIBA DILUTION RATIO FLUCTUATIONS DURING RADIATED 

IMMUNITY

The dilution ratio varied between 338 and -938 but only a smaller portion of the graph is 

shown for more detail. The external flowmeter on the end of the sample flow indicated that these 

rapid changes in flow were real and not just reported. The problem with the exhaust flow analog 

output signal first encountered during the bulk current injection testing was also observed during 

radiated immunity. 

The AVL system reported a supply voltage error in the range of 200 M Hz to 1 G Hz of 

horizontal r adiation. T he i nverter pow ered do wn a t 300 M Hz s hutting dow n t he s ystem 

completely.

5.4.3 Electrostatic Discharge

SAE t est J 1113/13 Electromagnetic Compatibility Measurement Procedure for Vehicle 

Components—Part 13: Immunity to Electrostatic Discharge was pe rformed to test t he P EMS 

response t o Electrostatic D ischarges ( ESDs) on surfaces and connectors. T he ex act pr ocedure 

used was taken from Region 2, Class B of the J1113/13 standard. Approximately 40 ESDs were 

-100

-80

-60

-40

-20

0

20

40

60

80

100

0 500 1000 1500 2000 2500 3000 3500

Dilution Ratio

Time (sec)

Measured Dilution Ratio Setpoint Dilution Ratio

50-70 MHz 

Vertical

120-150 MHz 

Vertical
REPORT 03.14936.12 126 of 174 

supplied to each PEMS at various connector por ts, exposed screws, and general surfaces. The 

test included both direct contact discharges as well as indirect discharges with the electrostatic 

discharge gun placed near the surface of interest. The ESD gun is shown in Figure 119. 

FIGURE 119. ELECTROSTATIC DISCHARGE SIMULATOR

The E SD g un w as c alibrated us ing a n e lectrostatic vol tmeter t o de liver 4000 vol ts pe r 

discharge. Neither the Horiba nor the AVL PEMS exhibited any susceptibility to the ESD tests. 

The Sensors unit shut off completely with an ESD near the auxiliary connector port to which an 

external f low m eter c onnects f or a udit pur poses. T his pr oblem w as c onfirmed w ith a s econd 

ESD. No issues possibly related to measurement accuracy were found with any of the PEMS.

5.4.4 Conducted Transients 

 

The response of the PEMS to voltage disturbances in the 12 volt power supply cable was 

checked using SAE test J1113/11 Immunity to Conducted Transients on Power Leads. The tests 

were c onducted us ing s pecifications f ound i n R egion 2, C lass B of t he J1113/11 pr otocol. A 

Schaffner NSG 5000 Automotive Electronics Test System was installed in between the 12 vol t 

power s upply a nd t he P EMS. T he S chaffner E lectronics T est S ystem de livered vol tage 

perturbations to the PEMS of varying magnitudes and durations. The voltage spikes ranged from 

-200 t o 100 vol ts a nd l asted a nywhere be tween 250 ns a nd 200 m s. The t ests i ncluded qui ck 

voltage recovery, slow voltage recovery, repeated voltage bursts, and load dump. The conducted 

transients t ests w ere no t pe rformed on the H oriba P EMS, because t he H oriba s ystem was 

intended for use with an external generator and does not use 12 volt power. The Schaffner Test 

System can be seen in Figure 120 with the Sensors PEMS and Figure 121 for the AVL PEMS. 
REPORT 03.14936.12 127 of 174 

FIGURE 120. SENSORS SETUP DURING CONDUCTED TRANSIENTS TESTING

FIGURE 121. AVL SETUP DURING CONDUCTED TRANSIENTS TESTING

The S ensors P EMS po wered dow n dur ing a -100 vol t s pike w ith qu ick r ecovery a nd 

again pow ered do wn w hen t he m agnitude of t he s pike w as reduced t o -50 vol ts a nd t hen -25 

volts. The slow recovery voltage spikes also caused the PEMS to power down with the shortest 

dwell time of 40 ms causing the unit to shut down. Longer dwell times were not tested. 

The quick recovery voltage spike caused the A VL inverter to power down, although it 

appeared t o be w orking pr operly w hen i t w as restarted. H owever, dur ing t he s low r ecovery 

voltage spikes, the inverter shut down again, started smoking and stopped working. The test was 

not repeated with another inverter to prevent further damage. 
REPORT 03.14936.12 128 of 174 

The f indings from t he EMI and R FI t esting were pr esented t o t he s teering c ommittee 

during t he M ay 20 th, 2009 i n S an A ntonio. T he s teering c ommittee de clined t o pur sue f urther 

testing to develop any EMI and RFI error surfaces. The steering committee requested that Horiba 

investigate t he i ssue of bul k c urrent i njection noi se i n t he analog ex haust f low cabl e and that 

Sensors investigate the issue of exhaust flow errors during radiated immunity testing.

5.5 Vibration Testing

Figures 122, 123, 124, 1 25, 126, and 127, show the setup for Sensors vibrational testing 

using di fferent c onfigurations. The vi bration testing w as condu cted as a s creening ex ercise 

similar to the m ethod us ed f or t he E MI a nd R FI t esting. T he P EMS w ere ope rated on H EPA 

filtered room air while being subjected to vibration. The response of the PEMS to the screening

exercise was used to determine whether further testing was needed to generate an error surface 

for vibration. Each PEMS was mounted on an Unholtze-Dickie Shaker Table which was capable 

of vibrating the PEMS on all three axes separately by adjusting the orientation of the PEMS. The 

system used a large table to vibrate horizontally. By rotating the orientation of the PEMS, this 

table w as a ble t o s imulate l ongitudinal a nd t ransverse hor izontal vi bration. T he s haker w as 

rotated into a vertical position and a smaller platform was attached to provide vertical vibration. 

Due t o c onsiderations f or non -road i n-use te sting, the s teering c ommittee r equested that th e 

PEMS a lso be r otated a t a 45 de gree i ncline o n e ach axis a s w ell. T he hor izontal vi bration 

platform was large enough to include all of the pieces of any one type of PEMS at a t ime. The 

vertical platform and 45 degree stand were each capable of installing only a single piece of the 

instrumentation at a time. 

FIGURE 122. SENSORS PEMS MOUNTED FOR TRANSVERSE HORIZONTAL 

VIBRATION
REPORT 03.14936.12 129 of 174 

FIGURE 123. SENSORS PEMS MOUNTED FOR LONGITUDINAL HORIZONTAL

VIBRATION

FIGURE 124. SENSORS PEMS MOUNTED FOR TRANSVERSE 45° VIBRATION

Direction of vibration

Direction of vibration
REPORT 03.14936.12 130 of 174 

FIGURE 125. SENSORS PEMS MOUNTED FOR LONGITUDINAL 45° VIBRATION

FIGURE 126. SENSORS PEMS MOUNTED FOR VERTICAL VIBRATION

Direction of vibration

Direction of vibration
REPORT 03.14936.12 131 of 174 

FIGURE 127. SENSORS PEMS MOUNTED FOR 45° VERTICAL VIBRATION

The pow er s pectral de nsity ( PSD) f rom t he M il S tandard 810, U S H ighway T ruck 

Vibration Exposure was used in this testing. To prevent damage to the PEMS, the PSD was only 

operated at 25%, 50%, and 75% of the energy specified in the Mil Standard 810. A fter a few 

tests at the 75% level, the energy levels were reduced to 10%, 25%, and 50% to maintain the 

integrity of t he i nstruments. In a ddition e ach e nergy l evel w as onl y t ested f or 5 m inutes t o 

minimize the chances of damaging the instruments. Figure 128 shows the PSD used for vertical 

vibration testing and Figure 130 shows the PSD used for horizontal testing. 

Figure 13 0 shows t he Horiba P EMS vi bration pos itions f or di fferent configurations. 

Because the OBS-TRPM included three separate boxes, some of the vibration testing had to be 

performed on i ndividual boxes. The DCS, HE, and MEEE were all able to fit on t he horizontal 

vibration table however only a single box could fit at a time on the 45 d egree angle stand. The 

DCS and MEEE were tested on t his stand; the HE box was not tested at 45 degrees, because it 

was felt that no errors would be detected in a box housing only a filter holder and a cyclone. A 

particle source would need to be present to detect problems in this portion of the system. The 

vertical vibration platform was able to hold both the DCS and the MEEE, so these two boxes 

were tested simultaneously while the HE was again not tested. Any piece of the system that was 

not being tested on t he vibration stand was positioned directly next to the stand so that all flow 

and electrical cables could still connect to the system normally. 

Direction of vibration
REPORT 03.14936.12 132 of 174 

FIGURE 128. POWER SPECTRAL DENSITY FOR VERTICAL VIBRATION TESTING

FIGURE 129. POWER SPECTRAL DENSITY FOR HORIZONTAL VIBRATION 

TESTING
REPORT 03.14936.12 133 of 174 

FIGURE 130. HORIBA PEMS VIBRATION POSITIONS

The H oriba P EMS ex perienced a m echanical f ailure dur ing t esting at t he 75% energy 

level of t he t ransverse horizontal vi bration. A bracket hol ding a f ilter inside t he M echanical 

Enclosure (ME) box broke causing a flow line to become blocked. The test was stopped and the 

bracket w as r eplaced with one f rom a nother uni t. T he t esting was r esumed unt il i t w as 

discovered t hat one o f the pr essure t ransducers us ed t o m easure t he f low w as not working 

properly c ausing a n error i n t he r eported f low. This pr oblem w as b elieved t o ha ve originated 

from t he br oken br acket be cause t he l ine t hat w as c losed of f w as a ttached t o t his pr essure 

transducer. The pressure transducer w as replaced with one from the other unit and the s ystem 

functioned properly. All future tests were conducted at 10%, 25%, and 50% energy for all PEMS 

to prevent further damage. The only other damage incurred by the Horiba was a rubber foot on 

the bottom of one of the boxes was sheared off. The measured dilution flow exhibited a higher 

degree of f luctuations d uring t he vi bration t esting t han w as nor mally observed although t he 

differences were not significant. 

The S ensors P EMS w as ope rated w ithout t he exhaust f low m eter and sample e lbow 

attached to the unit. As can be seen in Figure 127, the two pieces were removed to allow a HEPA 

filter t o be pl aced on t he i nlet. A lthough i t w ould ha ve be en de sirable t o l eave t hese pi eces 

attached it was c onsidered more i mportant t o provide cl ean air t o ensure a z ero particle l evel. 

The Sensors PEMS experienced no mechanical failures during vibration testing although several 

functionality i ssues di d oc cur. T he exhaust f low m easurement be came n oisier w ith vi bration 

although the magnitude of the noise was relatively small. In the EMI and RFI testing values as 

high as 1000 kg/hr were observed, but in vibration testing the exhaust flow never went above 50 

kg/hr. 
REPORT 03.14936.12 134 of 174 

The Sensors PEMS experienced problems maintaining its total flow rate during vibration 

in the vertical direction. This problem was related to the automatic drain valves on the moisture 

traps i n t he di lution f low l ine. T hese t raps a re designed t o ope n a utomatically w hen e nough 

downwards f orce i s a pplied f rom accumulated moisture i n t he reservoir; however t he ve rtical 

vibration w as c ausing t he va lves t o repeatedly o pen a llowing a por tion of t he f low t o e scape. 

Figure 131 shows an example of the total flow dropping while experiencing vertical vibration. 

FIGURE 131. SENSORS TOTAL FLOW DURING VIBRATION TESTING

It w as pos sible t o t est both of t he A VL boxes s imultaneously for t he horizontal a nd 

vertical vibration, however they were tested separately for the 45 degree angle vibration. Figure 

132 shows the AVL PEMS vibration testing using different configurations. 

No m echanical pr oblems w ere encountered w ith t he A VL uni t dur ing vi bration t esting 

although the measurement did exhibit vibration induced noise. The measured soot concentration 

fluctuated with increasing a mplitude a s the vi bration increased for a ll or ientations t ested. T he 

peak m easurement r ecorded w as about pl us a nd m inus 5 μ g/mol w ith m ore t ypical va lues 

swinging between plus and minus 2.5 μg/mol. A typical measurement is shown in Figure 133. 

0

2

4

6

8

10

200 400 600 800 1000 1200 1400 1600

Total Flow (slpm)

Time (sec)

None 10% 25% 50% None
REPORT 03.14936.12 135 of 174 

FIGURE 132. AVL PEMS VIBRATION POSITIONS

FIGURE 133. AVL SOOT MEASUREMENT NOISE DURING VIBRATION TESTING

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20

0 200 400 600 800 1000 1200

Reported Soot Concentration (mg/m3)

Time (sec)

Concentration dil. corrected [mg/m3] 30 Sec Avg

25% 50% 75% None
REPORT 03.14936.12 136 of 174 

This particular data was from the 45 degree angle longitudinal vibration. The noise in the 

measurement has no bi as indicating that it will average out over larger periods of time. The 30 

second average is s hown s ince t his i s l ikely t he m easurement error t hat w ould be obs erved 

during an NTE event. The maximum error of the 30 second average was -0.009 mg/m3 which is 

relatively i nsignificant b ased on P M c oncentrations e xpected a t t he t hreshold. H owever, i t is 

unclear from this testing whether the noise could be greater if a particle source was present. The 

results from the vibration testing were presented to the steering committee at the September 22nd, 

2009 meeting in Riverside. The steering committee declined to perform additional tests to create 

an error surface for vibration since the AVL s ystem was excluded from being an official PMPEMS.
REPORT 03.14936.12 137 of 174 

6.0 MODELING RESULTS

The main objective of this portion of the project was to use Monte Carlo techniques (e.g. 

random s ampling) in an e rror mode l to simulate the c ombined effects of a ll the a greed-upon 

sources of PEMS error incremental to lab error on the components of the brake-specific (BS) PM 

emissions. T his was accomplished by creating “error surfaces” for the Monte Carlo simulation 

to sample, based upon t he results of a variety of lab experiments. T he constructed model was 

simulated for thousands of trials (i.e., iterations) using data taken from a reference data set of 141 

unique N TE e vents. T he m odel r esults w ere u sed t o de termine t he br ake-specific a dditive 

measurement allowances for PM by three different calculation methods for three different PEMS 

model units. 

The error surfaces were generated from the results of each of the engine dynamometer 

and e nvironmental chamber laboratory tests. T he engine-lab-test error s urfaces cove red the 

domain of error versus the magnitude of the signal to which the error was to be applied (i.e., 1st

to 99 th percentile er ror vs. concentration, f low, t orque, e tc.). T he environmental-test e rror 

surfaces f or s hock and vi bration, a nd electromagnetic a nd radio f requency i nterference 

(EMI/RFI) was not included because no e rror surfaces were generated. The environmental test 

error surfaces for pressure and temperature were characteristically different because they covered 

the domain of the environmental-test cycle time versus the magnitude of the signal to which the 

error was to be applied (i.e., error at a selected time vs. concentration). 

6.1 Convergence Results from MC Runs 

This section contains a summary of the checks to determine if the convergence criteria 

were met for the simulation runs. Section 2.9 on Convergence and Number of Trials contains a 

detailed description of the convergence methodology and the procedures followed to check for 

convergence for the reference NTE event trials. This procedure was app lied to the simulation 

data obtained for each of the three PEMS units and all applicable calculation methods. 

Figure 134 through Figure 140 contain plots of the 90% confidence interval widths at the 

95th percentile delta differences (expressed as a p ercent of the BSPM emissions NTE threshold) 

versus the ideal BSPM emissions for the 141 individual reference NTE events. This is done for 

each of the three PEMS units and the applicable calculation methods. A summary of the results 

is given in Table 22. Of interest was whether or not the simulations converged within 1% of the 

threshold va lue. A s c an be s een i n t he pl ots, a majority of t he r eference N TE e vents di d not 

converge within 1% of the BSPM threshold. However, a majority did converge within 2% of the 

threshold. F or t he t hree P EMS uni ts a nd c alculation m ethods, t he m aximum pe rcent of t he 

confidence i nterval w idths r anged f rom 1.82% f or t he A VL M ethod 1 t o 2.76% f or S ensors 

Method 1. U pon e xamination of t he de lta BSPM di stributions f or t he va rious r eference N TE 

events, t hose t hat ha d a hi gh pe rcentage a bove t he B SPM t hreshold a t t he 95 th percentile 

generally had low input PM concentration levels.
REPORT 03.14936.12 138 of 174 

FIGURE 134. CONVERGENCE FOR AVL METHOD 1 AS A PERCENT OF BSPM 

THRESHOLD

FIGURE 135. CONVERGENCE FOR AVL METHOD 2 AS A PERCENT OF BSPM 

THRESHOLD

0.00

0.40

0.80

1.20

1.60

2.00

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

AVL Method 1

0.00

0.40

0.80

1.20

1.60

2.00

2.40

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

AVL Method 2
REPORT 03.14936.12 139 of 174 

FIGURE 136. CONVERGENCE FOR AVL METHOD 3 AS A PERCENT OF BSPM 

THRESHOLD

FIGURE 137. CONVERGENCE FOR HORIBA METHOD 1 AS A PERCENT OF BSPM 

THRESHOLD

0.00

0.40

0.80

1.20

1.60

2.00

2.40

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

AVL Method 3

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Horiba Method 1
REPORT 03.14936.12 140 of 174 

FIGURE 138. CONVERGENCE FOR HORIBA METHOD 2 AS A PERCENT OF BSPM 

THRESHOLD

FIGURE 139. CONVERGENCE FOR SENSORS METHOD 1 AS A PERCENT OF 

BSPM THRESHOLD

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Horiba Method 2

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

3.20

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Sensors Method 1
REPORT 03.14936.12 141 of 174 

FIGURE 140. CONVERGENCE FOR SENSORS METHOD 2 AS A PERCENT OF 

BSPM THRESHOLD

TABLE 22. SUMMARY OF NUMBER OF REFERENCE NTES MEETING 2% 

CONVERGENCE

PEMS Unit Method Min Max

No. NTEs within 

2% Convergence

% NTEs within 

2% Convergence

AVL

1 0.4467 1.8211 141 100%

2 0.4446 1.9766 141 100%

3 0.4240 1.9215 141 100%

Horiba 1 0.4158 2.5663 129 91%

2 0.4414 2.5447 131 93%

Sensors 1 0.7883 2.7664 125 87%

2 0.8613 2.6544 131 93%

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

0 0.01 0.02 0.03 0.04 0.05 0.06

% BSPM Threshold

Ideal BSPM, g/hp-hr

Sensors Method 2
REPORT 03.14936.12 142 of 174 

6.2 Sensitivity Based on Bias and Variance

 This section contains a summary of the error surfaces that contributed the most to the bias 

of the generated BS emissions. The sensitivity charts developed in Crystal Ball help identify the 

error surfaces (assumptions) that are sensitive to changes in variation with respect to their effect 

on the t hree de lta BS emissions. A nother t ype of s ensitivity e xamined i n t his s tudy w as 

concerned with the effects of potential “bias” in error surfaces and their effects on the forecast 

values. In or der t o s tudy these e ffects a ne w e rror s urface a ssumption was a dded t o t he MC 

Monte Carlo simulation model for each of the original 31 error surfaces. 

This assumption was sampled as a discrete binary distribution (i.e., on or off) during the 

simulation. F or e ach tr ial of the s imulation, 31 or iginal e rror s urfaces and 31 ‘ on/off’ error 

surfaces w ere s ampled according t o t heir de fined s ample di stribution. If t he ‘on/off’ e rror 

surface produced an ‘off’ condition, the delta emissions from that particular error surface were 

not added to the BS emissions computations for the BS emissions ‘with errors’. Similarly, if the 

‘on/off’ error surface produced an ‘on’ condition, the delta emissions from that particular error 

surface were added to the BS emissions calculations.

During every trial of the simulation, the exclusions due to the ‘off’ conditions resulted in 

various combinations of the error surface delta emissions being added to the BS emissions ‘with 

errors’ c omputations. Over t he c ourse of a MC s imulation w ith t housands of t rials, t he 

sensitivity of a particular error either ‘on’ or ‘off’ was assessed by examining the change in the 

forecast de lta em ission. T herefore, in a s ingle M C s imulation of a reference N TE ev ent 

sensitivities due to variance and/or bias were explored. 

Simulation results from the reference NTE events produced sensitivity values for all 95th 

percentile delta emissions by all three PEMS units and applicable calculation methods. Table 23 

through Table 2 9 summarize t he e rror s urfaces i n w hich e ither t he c ontribution-to-variance 

normalized sensitivity value or the ‘on/off’ bias check for the error surface was at least 5% in 

magnitude compared to all the other error surfaces. If the label in the error surface contains the 

words ‘Delta’ then it represents a che ck for bias; otherwise, the error surface indicates a ch eck 

for variance. Table 23 through Table 25 lists the sensitivity and bias descriptive statistics for the 

delta BSPM emissions for the AVL PEMS for Methods 1, 2 a nd 3, respectively. For all three 

methods, the largest mean normalized variance was from the bias effect due to error surface #1, 

SSPM. 
REPORT 03.14936.12 143 of 174 

TABLE 23. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR AVL BSPM METHOD 1 

Error 

Surface 

No. Error Surface

No. Ref 

NTE 

Events

Avg. Contribution

to Normalized

Variance, %

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 141 11.34 7.59 35.25

2 TR PM 4 7.00 6.42 7.34

20 SS Exhaust Flow 4 5.57 5.32 6.05

31 Torque Warm-up 10 -6.64 -11.22 -5.32

35 Torque Engine 

Manufacturer 1 -6.27 -6.27 -6.27

Delta Exhaust 

Flow Pulsation 3 5.78 5.56 5.99

Delta SS PM 141 -74.37 -83.67 -21.02

TABLE 24. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR AVL BSPM METHOD 2

Error 

Surface 

No. Error Surface

No. Ref 

NTE 

Events

Avg. Contribution

to Normalized

Variance, %

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 141 21.48 7.44 60.56

2 TR PM 38 9.20 5.53 12.31

31 Torque Warmup

8 -6.82 -10.77 -5.24

35 Torque Engine 

Manuf 1 -6.12 -6.12 -6.12

45 SS CO2 4 -6.89 -7.82 -5.32

Delta SS PM 141 -63.27 -83.77 -20.95

TABLE 25. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR AVL BSPM METHOD 3 

Error 

Surface 

No. Error Surface

No. Ref 

NTE 

Events

Avg. Contribution

to Normalized

Variance, %

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 141 20.91 7.71 60.50

2 TR PM 36 9.27 5.17 12.52

31 Torque Warmup

9 -6.62 -10.77 -5.14

35 Torque Engine 

Manuf 1 -6.12 -6.12 -6.12

45 SS CO2 4 -6.96 -7.89 -5.40

Delta SS PM 141 -64.04 -83.76 -20.93
REPORT 03.14936.12 144 of 174 

Table 26 and Table 27 list the s ensitivity a nd bias de scriptive s tatistics f or the de lta 

BSPM emissions for the Horiba PEMS for Methods 1 and 2, respectively. For both methods, the 

highest mean normalized variances were from the bias and variance due to error surface #1, SS 

PM. 

TABLE 26. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR HORIBA BSPM METHOD 1

Error 

Surface 

No. Error Surface

No. Ref 

NTE 

Events

Avg. Contribution

to Normalized

Variance, %

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 41.86 6.65 80.39

2 TR PM 10 5.84 5.04 7.18

20 SS Exhaust 

Flow 26 8.94 5.00 7.80

31 Torque Warmup

100 -8.34 -15.96 -5.08

35 Torque Engine 

Manuf 29 -5.72 -8.07 -5.00

Delta Exhaust 

Flow Pulsation 31 6.67 5.07 10.01

Delta SS PM 83 -32.05 -83.42 73.96

TABLE 27. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR HORIBA BSPM METHOD 2

Error 

Surface 

No. Error Surface

No. Ref 

NTE 

Events

Avg. Contribution

to Normalized

Variance, %

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 39.55 6.99 80.30

2 TR PM 10 5.89 5.00 7.42

31 Torque Warmup

95 -8.19 -15.04 -5.03

35 Torque Engine 

Manuf 24 -5.72 -7.65 -5.08

42 Fuel Rate 

Engine Manuf 7 5.52 5.07 6.34

45 SS CO2 54 -6.60 -9.83 -5.09

Delta SS PM 89 -42.60 -83.42 75.48
REPORT 03.14936.12 145 of 174 

Table 28 and Table 29 list the s ensitivity a nd bias de scriptive s tatistics f or the de lta 

BSPM emissions for the Sensors PEMS for Methods 1 a nd 2, respectively. F or both methods, 

the highest mean normalized variance was from the bias and variance due to error surface #1, 

SSPM.

TABLE 28. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR SENSORS BSPM METHOD 1

Error 

Surface 

No. Error Surface

No. Ref 

NTE 

Events

Avg. Contribution

to Normalized

Variance, %

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 46.92 6.57 78.61

2 TR PM 100 8.48 5.00 12.31

20 SS Exhaust Flow 8 6.11 8.06 7.56

31 Torque Warm-up 30 -8.28 -11.57 -5.02

35 Torque Engine 

Manuf 10 -5.60 -6.06 -5.14

Delta Exhaust 

Flow Pulsation 9 7.01 5.56 8.79

Delta SS PM 120 -31.19 -86.58 75.35

TABLE 29. ERROR SURFACE SENSITIVITY TO BIAS AND VARIANCE FOR 141 

REFERENCE NTE EVENTS FOR SENSORS BSPM METHOD 2

Error 

Surface 

No. Error Surface

No. Ref 

NTE 

Events

Avg. Contribution

to Normalized

Variance, %

Min 

Contribution,

%

Max 

Contribution,

%

1 SS PM 138 45.92 6.68 77.78

2 TR PM 101 8.48 5.03 12.52

31 Torque Warm-up 24 -7.84 -12.01 -5.08

35 Torque Engine 

Manuf 5 -5.61 -6.06 -5.01

42 Fuel Rate Engine 

Manuf 4 6.18 5.72 6.53

45 SS CO2 10 -6.84 -9.77 -5.15

Delta SS PM 127 -28.96 -86.67 75.75

The c ontribution t o normalized variance a nd bias s ensitivities f rom Table 23 through 

Table 29 are illustrated pictorially as box plots in Figure 141 to Figure 147 for BSPM by PEMS 

unit for Methods 1, 2 and 3. Only the error surfaces with at least 35 of the 141 r eference NTE 

events (1/4 of the events) are included as box plots. The mean normalized variance for each of 

the pl otted e rror s urfaces i s not ed b y a “ +” s ymbol i n t he box es. T he e rror s urface with t he 

largest mean normalized variance is plotted at the left of the chart. The error surface with the 

second largest mean normalized variance is plotted second from the left, and so on. Figure 142 

and Figure 143 demonstrate the high sensitivity to the negative bias for error surface #1, PM SS. 

Figure Figure146 and Figure 147 show a l arge variance ef fect due t o PM S S. Table 3 0 and 

Table 31 show a summary of the error surface sensitivity to bias and variance for the different 

PEMS using Method 1 a nd Method 2. T able 32 shows a similar summary using Method 3 for 

the AVL PEMS only. 
REPORT 03.14936.12 146 of 174 

FIGURE 141. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR AVL BSPM METHOD 1

FIGURE 142. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR AVL METHOD 2
REPORT 03.14936.12 147 of 174 

FIGURE 143. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR AVL METHOD 3

FIGURE 144. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR HORIBA BSPM METHOD 1
REPORT 03.14936.12 148 of 174 

FIGURE 145. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR HORIBA BSPM METHOD 2

FIGURE 146. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS 

AND VARIANCE FOR SENSORS BSPM METHOD 1
REPORT 03.14936.12 149 of 174 

FIGURE147. BOX PLOT OF ERROR SURFACE SENSITIVITY BASED ON BIAS AND 

VARIANCE FOR SENSORS BSPM METHOD 2

TABLE 30 SUMMARY OF ERROR SURFACE SENSITIVITIES TO BIAS AND 

VARIANCE FOR BSPM METHOD 1

Method 1

Error

Surface

No. Error Surface

AVL Horiba Sensors

No. NTE 

Events 

Avg 

Contribution 

to Normalized 

Variance, %

No. NTE 

Events 

Avg 

Contribution to 

Normalized 

Variance, %

No. NTE 

Events 

Avg 

Contribution to 

Normalized 

Variance, %

1 SS PM 141 11.34 138 41.86 138 46.92

2 TR PM 4 7.00 10 5.84 100 8.48

20

SS Exhaust 

Flow 4 5.57 26 8.94 8 6.11

31

Torque Warmup 10 -6.64 100 -8.34 30 -8.28

35

Torque Engine 

Manuf 1 -6.27 29 -5.72 10 -5.60

Delta Exhaust 

Flow Pulsation 3 5.78 31 6.67 9 7.01

Delta SS PM 141 -74.37 83 -32.05 120 -31.19
REPORT 03.14936.12 150 of 174 

TABLE 31. SUMMARY OF ERROR SURFACE SENSITIVE TO BIAS AND 

VARIANCE FOR BSPM METHOD 2

Method 2

Error

Surface

No. Error Surface

AVL Horiba Sensors

No. NTE 

Events

Avg 

Contribution 

to Normalized 

Variance, %

No. NTE 

Events

Avg 

Contribution to 

Normalized 

Variance, %

No. NTE 

Events

Avg 

Contribution to 

Normalized 

Variance, %

1 SS PM 141 21.48 138 39.55 138 45.92

2 TR PM 38 9.20 10 5.89 101 8.48

31

Torque Warmup 8 -6.82 95 -8.19 24 -7.84

35

Torque Engine 

Manuf 1 -6.12 24 -5.72 5 -5.61

42

Fuel Rate 

Engine Manuf 7 5.52 4 6.18

45 SS CO2 4 -6.89 54 -6.60 10 -6.84

Delta SS PM 141 -63.27 89 -42.60 127 -28.96

TABLE 32. SUMMARY OF ERROR SURFACE SENSITIVE TO BIAS AND 

VARIANCE FOR BSPM METHOD 3

Method 3

Error

Surface

No. Error Surface

AVL

No. NTE 

Events

Avg Contribution to 

Normalized

Variance, %

1 SS PM 141 20.91

2 TR PM 36 9.27

31 Torque Warm-up 9 -6.62

35

Torque Engine 

Manuf 1 -6.12

45 SS CO2 4 -6.96

Delta SS PM 141 -64.04

6.3 Validation Results

This section contains a summary of the model validation results; Section 0 on Validation

contains a m ore d etailed de scription of t he validation m ethodology ut ilized bot h i n t he 

simulation and in the on-road data collection efforts.

During the Monte Carlo simulation of the 141 r eference NTE events some of the error 

surfaces w ere e xcluded i n t he c omputation of t he B S e missions ‘ with e rrors’ s o t hat t he 

simulation r epresented c onditions us ed i n c ollecting t he on -road data. T he er ror s urfaces 

excluded were torque errors (Nos. 29-32, 34, 35), fuel rate engine manufacturers (#42), dynamic 
REPORT 03.14936.12 151 of 174 

speed (#43) and dynamic fuel rate (#44). For each reference NTE event, the difference in BSPM 

emissions was computed as 

delta BSPM = BSPM with “Validation error” – “Ideal” BSPM.

These delta BSPM em issions were com puted for each of t he t hree P EMS uni ts and all 

applicable calculation methods. T he 5 th, 50 th and 95 th percentiles w ere id entified from the 

distributions of t he de lta B SPM e missions dur ing t he M onte C arlo simulation us ing t he 

validation error surfaces only. Figure 148 t hrough Figure 150 depict the validation percentiles 

for the AVL PEMS unit for methods 1, 2 and 3, respectively. Similar validation plots for the 

Horiba PEMS unit are illustrated in Figure 151 and Figure 152 for methods 1 and 2, respectively. 

Sensors PEMS validation plots for methods 1 and 2 are shown in Figure 153 through Figure 154. 

FIGURE 148. VALIDATION PERCENTILES FOR THE 141 REFERENCE NTE 

EVENTS FOR AVL METHOD 1

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation AVL 1 5th Validation AVL 1 50th Validation AVL 1 95th
REPORT 03.14936.12 152 of 174 

FIGURE 149. VALIDATION PERCENTILES FOR THE 141 REFERENCE NTE 

EVENTS FOR AVL METHOD 2

FIGURE 150. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR AVL METHOD 3

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation AVL 2 5th Validation AVL 2 50th Validation AVL 2 95th

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation AVL 3 5th Validation AVL 3 50th Validation AVL 3 95th
REPORT 03.14936.12 153 of 174 

FIGURE 151. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR HORIBA METHOD 1

FIGURE 152. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR HORIBA METHOD 2

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.020

0.025

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Horiba 1 5th Validation Horiba 1 50th Validation Horiba 1 95th

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.020

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Horiba 2 5th Validation Horiba 2 50th Validation Horiba 2 95th
REPORT 03.14936.12 154 of 174 

FIGURE 153. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR SENSORS METHOD 1

FIGURE 154. VALIDATION PERCENTILES FOR 141 REFERENCE NTE EVENTS 

FOR SENSORS METHOD 2

-0.035

-0.030

-0.025

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Sensors 1 5th Validation Sensors 1 50th Validation Sensors 1 95th

-0.035

-0.030

-0.025

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.00 0.01 0.02 0.03 0.04 0.05 0.06

Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

 

Validation Sensors 2 5th Validation Sensors 2 50th Validation Sensors 2 95th
REPORT 03.14936.12 155 of 174 

The 5th and 95th percentiles of the validation delta BSPM were separately fit for the AVL 

and S ensors P EMS uni ts us ing a s imple l inear regression m odel. H owever, t he c riteria for 

accepting the linear fits were not met for any of the calculations methods for these two PEMS 

units. T hus, loess regression fits were used to determine the best functional representation for 

the 5th and 95th delta BSPM based on the validation simulation modeling. These loess fits for the 

95th and 5th percentiles for the Sensors unit methods 1 and 2 can be found in Figure 155 through 

Figure 158 , respectively. T he l oess f its f or t he 95 th and 5 th percentiles f or t he A VL uni ts 

methods 1, 2 a nd 3 c an be f ound i n Figure 159 t hrough F igure 164 , respectively. T he l oess 

smoothing parameters for the regression fits are listed in Table 33. 

TABLE 33. LOESS SMOOTHING PARAMETERS FOR VALIDATION PERCENTILES

PEMS Method 5th Percentile 95th Percentile

Sensors 1 0.290 0.570

2 0.290 0.570

AVL

1 0.294 0.755

2 0.294 0.777

3 0.294 0.777

FIGURE 155. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

SENSORS METHOD 1
REPORT 03.14936.12 156 of 174 

FIGURE 156. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

SENSORS METHOD 1

FIGURE 157. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

SENSORS METHOD 2
REPORT 03.14936.12 157 of 174 

FIGURE 158. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

SENSORS METHOD 2

FIGURE 159. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR

AVL METHOD 1
REPORT 03.14936.12 158 of 174 

FIGURE 160. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 1

FIGURE 161. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 2
REPORT 03.14936.12 159 of 174 

FIGURE 162. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 2

FIGURE 163. VALIDATION 95TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 3
REPORT 03.14936.12 160 of 174 

FIGURE 164. VALIDATION 5TH PERCENTILE BSPM DELTAS LOESS FIT FOR 

AVL METHOD 3

6.4 Measurement Error Allowance Results 

This section contains a summary of the measurement error allowance results using both a 

regression method and a median method to determine the measurement allowance. Section 2.12 

on Measurement Allowance contains a de tailed de scription of t he m ethodology f ollowed i n 

determining these values. This procedure was applied to the simulation data for all 141 reference 

NTE events obtained for all three calculation methods for the AVL PEMS and for calculations 

methods 1 and 2 for the Horiba and Sensors PEMS. 

Figure65 contains a regression plot of the 95th percentile delta BSPM values versus the 

Ideal BSPM values for the 141 reference NTE events for AVL Method 1. Included in the plot is 

the equation for the fitted regression line, and the R-square (R2) value and root mean square error 

(RMSE) va lue f or t he r egression f it. T he t wo symbols i n t he pl ot r epresent r eference N TE 

events where there was a dominant bias effect due to the SSPM error surface (diamond symbol) 

or there was a dominant variance effect due to the SSPM error surface (square symbol). The Rsquare va lue i ndicates t hat 47.95% of t he va riation in t he 95 th percentile B SPM va lues i s 

explained by the ideal BSPM values for the AVL Method 1 da ta. T he RMSE value of 0.0008 

displays t he s ize of t he e stimated s tandard de viation of t he pr edicted 95th percentile B SPM 

values.
REPORT 03.14936.12 161 of 174 

Table 34 includes a comparison of the results of the regression method based on Figure 

165 and the median method as described in the Section 0 on Measurement Allowance. U nder 

the heading of “Regression Method” in the table, it is shown that only the R-square criterion was 

not m et b y t he da ta. T hus, t he M edian M ethod m ust be us ed. Under t he he ading “ Median 

Method” in the table, the measurement error at the BSPM threshold, based on using the median 

of t he 141 95 th percentile de lta BSPM va lues, i s 0.661% w hen e xpressed a s a p ercent of t he 

threshold of 0.02 g/hp-hr. 

FIGURE165. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR AVL METHOD 1

TABLE 34. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR AVL METHOD 1

Regression Method Median Method

R2 0.4795 Did Not Meet 

Criteria

RMSE (SEE) 0.0008 Met Criteria

5% Median Ideal 0.0191007

Predicted 95th % Delta

at Threshold 0.0003399 Median 95th % Delta 0.0001322 

Measurement Error @ 

Threshold = 0.02 1.6993% Measurement Error @ Threshold = 0.02 0.661% 

y = 0.116x - 0.002

R2 = 0.4795

RMSE = 0.0008

-0.0020

-0.0010

0.0000

0.0010

0.0020

0.0030

0.0040

0.0050

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM g/hp-hr

Ideal BSPM, g/hp-hr

AVL Method 1

Delta SSPM ic SSPM
REPORT 03.14936.12 162 of 174 

Figure 166 contains a regression plot of the 95th percentile delta BSPM values versus the 

Ideal BSPM values for the 141 r eference NTE events for AVL Method 2. T he R-square value 

indicates t hat 46.02% of t he va riation i n t he 95 th percentile B SPM va lues i s e xplained by t he 

Ideal BSPM values for the AVL Method 2 data. The RMSE value is 0.0008. 

Table 35 includes a comparison of the results of the regression method based on Figure 

166 and the median method. Under the heading of “Regression Method” in the table, it is shown 

that the R -square c riterion for using t his m ethod i s not m et b y t he da ta. T hus, t he M edian 

Method must be used. Under the heading “Median Method” in the table, the measurement error 

at the BSPM threshold, based on using the median of the 141 95th percentile delta BSPM values, 

is -2.375% when expressed as a percent of the threshold value of 0.02. 

FIGURE 166. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR AVL METHOD 2

TABLE 35. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR AVL METHOD 2

Regression Method Median Method

R 0.4602 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0008 Met Criteria

5% Median Ideal 0.0191007

Predicted 95th

-0.0002124 % Delta at 

Threshold

Median 95 th

-0.0004751 % 

Delta

Measurement E rror @ 

Threshold = 0.02 -1.0618% Measurement E rror @ Threshold = 0.02 -2.375% 

y = 0.0996x - 0.0022

R2 = 0.4602

RMSE = 0.0008

-0.0020

-0.0010

0.0000

0.0010

0.0020

0.0030

0.0040

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

AVL Method 2

Delta SSPM ic SSPM
REPORT 03.14936.12 163 of 174 

Figure 167 contains a regression plot of the 95th percentile delta BSPM values versus the 

Ideal BSPM values for the 141 r eference NTE events for AVL Method 3. T he R-square value 

indicates t hat 46.20% of t he va riation i n t he 95 th percentile B SPM va lues i s e xplained by t he 

Ideal BSPM values for the AVL Method 3 data. The RMSE value is 0.0008. 

Table 36 i ncludes a comparison of the results of the regression method based on Figure 

167 and the median method. Under the heading of “Regression Method” in the table, it is shown 

that the R -square c riterion for u sing t his m ethod i s not m et b y t he da ta. T hus, t he M edian 

Method must be used. Under the heading “Median Method” in the table, the measurement error 

at the BSPM threshold, based on using the median of the 141 95th percentile delta BSPM values, 

is -2.383% when expressed as a percent of the threshold value of 0.02. 

FIGURE167. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR AVL METHOD 3

TABLE 36. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR AVL METHOD 3

Regression Method Median Method

R 0.4620 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0008 Met Criteria

5% Median Ideal 0.0191007

Predicted 95th

-0.0002118 % Delta at 

Threshold 

Median 95 th

-0.0004766 % 

Delta

Measurement E rror @ 

Threshold = 0.02 -1.0592% Measurement E rror @ Threshold = 0.02 -2.383% 

y = 0.1001x - 0.0022

R2 = 0.462

RMSE = 0.0008

-0.0020

-0.0010

0.0000

0.0010

0.0020

0.0030

0.0040

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

AVL Method 3

Delta SSPM ic SSPM
REPORT 03.14936.12 164 of 174 

Figure 168 contains a regression plot of the 95th percentile delta BSPM values versus the 

Ideal BSPM values for the 141 reference NTE events for Horiba Method 1. The R-square value 

indicates t hat 26.57% of t he va riation i n t he 95 th percentile B SPM va lues i s e xplained by t he 

Ideal BSPM values for the Horiba Method 1 data. The RMSE value is 0.0041. 

Table 37 includes a comparison of the results of the regression method based on Figure 

168 and the median method. Under the heading of “Regression Method” in the table, it is shown 

that the R-square and the RMSE criteria for using this method were not met by the data. Thus, 

the M edian M ethod m ust be us ed. U nder t he heading “ Median M ethod” i n t he t able, t he 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 54.379 % when expressed as a percent of the threshold value of 0.02. 

FIGURE168. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR HORIBA METHOD 1

TABLE 37. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR HORIBA METHOD 1

Regression Method Median Method

R 0.2657 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0041 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0102566 % Delta at 

Threshold 

Median 95 th

0.0108759 % 

Delta

Measurement E rror @ 

Threshold = 0.02 51.2831% Measurement E rror @ Threshold = 0.02 54.379% 

y = -0.3555x + 0.0174

R2 = 0.2657

RMSE = 0.0041

-0.0050

0.0000

0.0050

0.0100

0.0150

0.0200

0.0250

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Horiba Method 1

Delta SSPM ic SSPM
REPORT 03.14936.12 165 of 174 

Figure 169 contains a regression plot of the 95th percentile delta BSPM values versus the 

Ideal BSPM values for the 141 reference NTE events for Horiba Method 2. The R-square value 

indicates that 25.97 % of the variation in the 95th percentile BSPM values is explained by the 

Ideal BSPM values for the Horiba Method 2 data. The RMSE value is 0.0041. 

Table 38 includes a comparison of the results of the regression method based on Figure 

169 and the median method. Under the heading of “Regression Method” in the table, it is shown 

that the R-square and the RMSE criteria for using this method were not met by the data. Thus, 

the M edian M ethod m ust be us ed. U nder t he heading “ Median M ethod” i n t he t able, t he 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 50.079 % when expressed as a percent of the threshold value of 0.02. 

FIGURE169. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR HORIBA METHOD 2

TABLE 38. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR HORIBA METHOD 2

Regression Method Median Method

R 0.2597 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0041 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0094282 % Delta at 

Threshold

Median 95 th

0.0100158 % 

Delta

Measurement E rror @ 

Threshold = 0.02 47.1408% Measurement E rror @ Threshold = 0.02 50.079% 

y = -0.3515x + 0.0165

R2 = 0.2597

RMSE = 0.0041

-0.0050

0.0000

0.0050

0.0100

0.0150

0.0200

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Horiba Method 2

Delta SSPM ic SSPM
REPORT 03.14936.12 166 of 174 

Figure 170 contains a regression plot of the 95th percentile delta BSPM values versus the 

Ideal BSPM values for the 141 reference NTE events for Sensors Method 1. The R-square value 

indicates that 45.49 % of the variation in the 95th percentile BSPM values is explained by the 

Ideal BSPM values for the Sensors Method 1 data. The RMSE value is 0.0029. 

Table 39 i ncludes a comparison of the results of the regression method based on Figure 

170 and the median method. Under the heading of “Regression Method” in the table, it is shown 

that the R-square and the RMSE criteria for using this method were not met by the data. Thus, 

the M edian M ethod m ust be us ed. U nder t he heading “ Median M ethod” i n t he t able, t he 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 34.361 % when expressed as a percent of the threshold value of 0.02. 

FIGURE 170. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR SENSORS METHOD 1

TABLE 39. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR SENSORS METHOD 1

Regression Method Median Method

R2 0.4549 Did Not 

Meet Criteria

RMSE (SEE) 0.0029 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th % Delta 

at Threshold 0.0066785 Median 95th % Delta 0.00687227 

Measurement Error @ 

Threshold = 0.02 33.3924% Measurement Error @ Threshold = 0.02 34.361% 

y = -0.3866x + 0.0144

R2 = 0.4549

RMSE = 0.0029

-0.0150

-0.0100

-0.0050

0.0000

0.0050

0.0100

0.0150

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Sensors Method 1

Delta SSPM ic SSPM
REPORT 03.14936.12 167 of 174 

Figure 171 contains a regression plot of the 95th percentile delta BSPM values versus the 

Ideal BSPM values for the 141 reference NTE events for Sensors Method 2. The R-square value 

indicates that 45.03 % of the variation in the 95th percentile BSPM values is explained by the 

Ideal BSPM values for the Sensors Method 1 data. The RMSE value is 0.0030. 

Table 40 includes a comparison of the results of the regression method based on Figure 

171 and the median method. Under the heading of “Regression Method” in the table, it is shown 

that the R-square and the RMSE criteria for using this method were not met by the data. Thus, 

the M edian M ethod m ust be us ed. U nder t he heading “ Median M ethod” i n t he t able, t he 

measurement error at the BSPM threshold, based on using the median of the 141 95th percentile 

delta BSPM values, is 30.285 % when expressed as a percent of the threshold value of 0.02. 

FIGURE 171. REGRESSION PLOT OF 95TH PERCENTILE DELTA BSPM VERSUS 

IDEAL BSPM FOR SENSORS METHOD 2

TABLE 40. MEASUREMENT ERROR AT THRESHOLD FOR BSPM USING 

REGRESSION AND MEDIAN METHODS FOR SENSORS METHOD 2

Regression Method Median Method

R 0.4503 2 Did Not 

Meet Criteria

RMSE (SEE) 0.0030 Did Not 

Meet Criteria

5% Median Ideal 0.0191007

Predicted 95th

0.0059333 % Delta at 

Threshold

Median 95 th

0.0060569 % 

Delta

Measurement E rror @ 

Threshold = 0.02 29.6663% Measurement E rror @ Threshold = 0.02 30.285% 

y = -0.3875x + 0.0137

R2 = 0.4503

RMSE = 0.0030

-0.0150

-0.0100

-0.0050

0.0000

0.0050

0.0100

0.0150

0 0.01 0.02 0.03 0.04 0.05 0.06

95th % Delta BSPM, g/hp-hr

Ideal BSPM, g/hp-hr

Sensors Method 2

Delta SSPM ic SSPM
REPORT 03.14936.12 168 of 174 

Table 41 contains a s ummary of t he m easurement e rror va lues c ontained i n Table 3 4 

through Table 40. The values are categorized by PEMS unit and by calculation method. 

TABLE 41. BSPM MEASUREMENT ERROR IN PERCENT OF NTE THRESHOLD BY 

PEMS AND CALCULATION METHOD

Measurement Errors (%) at Respective NTE Threshold

PEMS

Method 1

Exhaust Flow 

Torque-Speed

Method 2

Exhaust and Fuel

Flow Torque-Speed

Method 3

Fuel Flow

Torque-Speed

AVL 0.661 -2.375 -2.383

Horiba 54.379 50.079 n/a

Sensors 34.361 30.285 n/a

Table 42 includes i n the m easurement al lowance s elected based on the m inimum 

normalized PM. The AVL w as not used in the measurement allowance determination because 

the AVL at the start of the program was not accepted as an official PEMS, and the measurement 

Steering Committee had decided that the measurement allowance would only be based on t he 

Sensors or the Horiba PEMS. 

TABLE 42. MEASUREMENT ALLOWANCE AT NTE THRESHOLD BY EMISSIONS 

FOR METHOD 2 

PEMS

Method 2 

Measurement 

Error %

NTE 

Threshold 

g/hp-hr

Measurement 

Allowance, 

g/hp-hr

Sensors 30.285 0.02 0.00605

On-road P M e missions were gathered f rom s elected routes dr iven to collect e missions 

data with a CE-CERT trailer and a PEMS installed on the tractor pulling the trailing. For each 

on-road NTE event, a delta BSPM emissions value was computed as follows: 

Delta BSPM = PEMS BSPM – CE-CERT BSPM.

These differences were computed for the BSPM emissions for each PEMS unit tested inuse. The in-use BSPM was computed using Method 1 and 2 for the Sensors PPMD and Method 

1, 2, a nd 3 f or A VL M SS. The i n-use de lta B SPM em issions cal culated for t he AVL PE MS 

using methods 1, 2, a nd 3. CE-CERT validation data were produced without any diesel particle 

filter ( DPF) active regeneration (referred to as “no regen”). (referred to a s ” )informational 

purposes active AVL methods 1, 2 and 3 including three individual units (#2, #3 and #4) and 271 

NTE e vents. T his da ta s et w as computed a s “no r egen”. T he s econd P EMS uni t t ested f or 

validation w as t he S ensors. T he on -road delta BSPM em issions w ere calculated f or S ensors 

methods 1 and 2 also using three individual units in the “no regen” scenario and resulted in 217 

NTE events.
REPORT 03.14936.12 169 of 174 

The va lidation pl ots f or t he S ensors a nd t he A VL s ystems a re s hown in Figures 172 

through 176. The y-axis scale on each figure was intended to make the best representation of the 

data relative to the validation lines shown on each plot. Because Method 1 depends strongly on 

exhaust flow, and some concerns were raised about the accuracy of exhaust flow measurement 

during CE-CERT testing, results reported using Method 1 m ight not be accurate. Details about 

exhaust flow measurement are expected to be part of CE-CERT Final Report on PM-PEMS InUse Validation Testing. 

The loess regression fits in Figures 176 t hrough 176 and the Sensors CE-CERT BSPM 

differences col lected on -road were pl otted i n order t o determine i f t he s imulation m ethod 

validated. If t he num ber of C E-CERT de lta BSPM va lues doe s not e xceed 10% o f t he t otal 

number of on-road NTE events collected, then the simulation method would be considered valid. 

Figure 172 represents the validation plot for the BSPM method 1 analysis for Sensors. Note that 

7 of the 217 on -road NTE events were either below or above the range of the ideal BSPM and 

were excluded from the validation percentage calculation. T herefore, 68 of the 210 C E-CERT 

NTE e vents (32.38%) fell be low t he s imulation m odel 5 th percentile ba sed on the l oess 

regression. Thus, the model was not considered valid for the BSPM Sensors Method 1. 

Figure 173 represents the validation plot for the BSPM method 2 a nalysis for Sensors. 

Note that again 7 of the 217 on -road NTE events were either below or above the range of the 

ideal BSPM and were excluded from the validation percentage calculation. Therefore, 71 of the 

210 CE-CERT NTE events (33.81%) fell above the simulation model 95th percentile or below 

the s imulation model 5 th percentile ba sed on t he l oess r egressions. T hus, t he m odel w as not 

considered valid for the BSPM Sensors Method 2. 

Figure 174 represents the validation plot for the BSPM method 1 analysis for AVL with 

‘no regen’. In this case, 8 of the 271 on-road NTE events were either below or above the range 

of the ideal BSPM and were excluded from the validation percentage calculation. Therefore, 49 

of the 263 C E-CERT NTE events (18.63%) fell above the simulation model 95th percentile or 

below the simulation model 5th percentile based on t he loess regressions. T hus, the model was 

not considered valid for the BSPM AVL Method 1.

Figure 175 represents the validation plot for the BSPM method 2 analysis for AVL with 

‘no regen’. Again in this case 8 of the 271 on-road NTE events were either below or above the 

range o f t he i deal B SPM and were ex cluded f rom t he v alidation percentage calculation. 

Therefore, 30 of the 263 CE-CERT NTE events (11.41%) fell above the simulation model 95th

percentile or below the simulation model 5th percentile based on the loess regressions. Thus, the 

model was not considered valid for the BSPM AVL Method 2. 

Figure 176 represents the validation plot for the BSPM method 3 analysis for AVL with 

‘no regen’. Again in this case 8 of the 271 on-road NTE events were either below or above the 

range o f t he i deal B SPM and were ex cluded f rom t he v alidation percentage calculation. 

Therefore, 26 of the 263 CE-CERT NTE events (9.89%) fell above the simulation model 95th

percentile or below the simulation model 5th percentile based on the loess regressions. Thus, the 

model was considered valid for the BSPM AVL Method 3 since the number of CE-CERT NTE 

events outside the 5th and 95th percentile loess regression was less than 10%.
REPORT 03.14936.12 170 of 174 

FIGURE 172. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM SENSORS METHOD 1 WITH NO REGEN

FIGURE 173. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM SENSORS METHOD 2 WITH NO REGEN
REPORT 03.14936.12 171 of 174 

FIGURE 174. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 1 WITH NO REGEN

FIGURE 175. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 2 WITH NO REGEN
REPORT 03.14936.12 172 of 174 

FIGURE 176. VALIDATION ON-ROAD AND REGRESSION FUNCTIONS BASED ON 

THE SIMULATION MODEL FOR BSPM AVL METHOD 3 WITH NO REGEN

Table 43 summarizes the model validation results. Only the AVL Method 3 pa ssed the 

model validation. 

TABLE 43. SUMMARY OF BSPM MODEL VALIDATION RESULTS

PEMS Unit

Method 1

Exhaust Flow 

Torque-Speed

Method 2

Exhaust and Fuel

Flow Torque-Speed

Method 3

Fuel Flow

Torque-Speed

Sensors “no regen” No No No

AVL “no regen” No No Yes
REPORT 03.14936.12 173 of 174 

7.0 SUMMARY

A s eries of en gine ex periments and environmental t ests w ere p erformed on three P MPEMS that included the Sensors PPMD, Horiba TRPM, and AVL MSS. The Sensors PPMD and 

Horiba TRPM were treated as official PEMS, and only those were used for official results and 

determination of next steps. The AVL MSS were used in conjunction with the Sensors PPMD 

based on an agreement reached between Sensors and AVL. Error surfaces were developed based 

on e xperimental w ork t o de termine P M-PEMS bi as a nd pr ecision e rrors us ing M onte C arlo 

simulation model. The o utput of the mod el w as to determine th e e rror d istribution at a s et of 

reference N TE ev ents, compared t o their i deal va lue. The P M-PEMS w ith t he l owest 95 th

percentile error that is greater than zero was selected for in-use validation testing of the model.

The P M-PEMS t hat w as s elected for i n-use va lidation w as t he S ensors P PMD. T he 

PPMD produced a 95th percentile measurement allowance error of 0.006 g/hp-hr at a threshold 

NTE limit of 0.02 g /hp-hr using Method 2, c ompared to the 0.01 g /hp-hr that was produced by 

the Horiba TRPM. As for the AVL MSS, the instrument produced zero measurement allowance, 

but its measurement allowance value was not officially used because the MSS only measures the 

carbon fraction of PM, compared to the required total (solid plus volatile) PM measured by the 

other two PM-PEMS. However, because the AVL MSS was used in conjunction with the PPMD, 

the SC a greed t o i nclude i t dur ing i n-use va lidation testing. D ue to funding limita tion, the 

Horiba TRPM was not included in in-use validation testing.

Based on i n-use v alidation t esting, t he S ensors P PMD f ailed va lidation be cause 3 2 

percent and 34 percent of the data produced in-use were below the 5th percentile of the validation 

window, using Method 1 and Method 2, respectively. The SC agreed during the development of 

the Test Plan that less or equal 10 percent (≤ 10%) of the data are allowed to be outside the 

validation window to pass validation. 

As for the AVL MSS, it passed validation using Method 3 by having 9.89 percent of the 

data outside the v alidation window, with the majority of these data being hi gher than the 9 5th

percentile. Method 2 failed by two percentage points and Method 1 failed by 8 percentage points. 

Because the MSS using Method 3 passed validation and funding run out to do any further 

work with the Sensors PPMD and/or to perform in-use validation testing with the Horiba TRPM, 

the SC concluded the measurement allowance program. The SC also accepted the measurement 

allowance ba sed on t he S ensors P PMD a s t he f inal P M-PEMS m easurement al lowance for 

Methods 1, 2, and 3 
REPORT 03.14936.12 174 of 174 

8.0 REFERENCES

1. “Electronic C ode of Federal R egulations, T itle 4 0 P rotection of t he E nvironment, V ol. 15 

Part 1065,” http://www.ecfr.gpoaccess.gov, 2009. 

2. “Determination of PEMS Measurement Allowance for Gaseous Emissions Regulated Under 

the H eavy-Duty I n-Use T esting P rogram: R evised Final R eport,” EPA420-R-08-005, 

February 2008. 

3. Conover, W.J., Practical Nonparametric Statistics, John Wiley & Sons, 1971 

4. Cleveland, W .S., “ Robust Locally-Weighted Regression a nd S moothing S catterplots,” 

Journal of the American Statistical Association, 1979, 74, pp. 829-836. 

5. Hurviuch, C .M., S imonoff, J .S., a nd T sai, C .L., “Smoothing P arameters S election i n 

Nonparametric R egression U sing a nd Improved A kaike Information C riterion,” J ournal of 

the Royal Statistical Society B, 1998, 60, pp.271-293. 

6. Stratmann, F., Otto, E., and Fissan, H., “Thermophoretical and Diffusional Particle Transport 

in a Cooled Laminar Tube Flow.” Journal of Aerosol Science, March 28th, 1994, vol. 25; pp. 

1359-1365. 

7. Liu, B .Y.H and D .Y.H. P ui, “ Equilibrium B ipolar C harge D istribution of A erosols,” J . 

Colloid Interface Science, Vol. 49, 1974. 
REPORT 03.14936.12

APPENDIX A

TEST PLAN TO DETERMINE PEMS MEASUREMENT ALLOWANCE 

FOR THE PM EMISSIONS REGULATED UNDER THE 

MANUFACTURER-RUN HEAVY-DUTY DIESEL ENGINE IN-USE 

TESTING PROGRAM

DEVELOPED BY:

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY,

CALIFORNIA AIR RESOURCES BOARD, AND

ENGINE MANUFACTURERS ASSOCIATION

NOVEMBER 11, 2008
REPORT 03.14936.12 A-1 

EXECUTIVE SUMMARY

This test plan sets forth the agreed upon pr ocesses and methodologies to be utilized to develop 

additive, br ake-specific, da ta-driven measurement allowance for PM emissions me asured b y 

PEMS as required under the HDIUT regulatory program.

As detailed in this test plan, there is a clear consensus on what components of measurement error 

are i ntended to be covered by t he m easurement allowance. N amely, t he allowance is to b e 

calculated in a manner that subtracts lab error from PEMS error. Specifically, utilizing Part 1065 

compliant emissions measurement systems and procedures for both the lab and PEMS, the lab 

error associated with measuring heavy-duty engine emissions at stabilized steady-state test points

within the NTE zone, will be subtracted from the PEMS error associated with measuring heavyduty e ngine e missions utilizing P EMS ove r events unde r a br oad r ange of environmental 

conditions. This subtraction will yield “PEMS minus laboratory” measurement allowance. The 

experimental methods and procedures specified in this test plan for determining, modeling, and 

comparing e ach of t he various c omponents of measurement error a re designed t o generate 

statistically robust data-driven measurement allowance for the PM emissions. 

Successful completion of this test plan is part of the resolution of a 2001 suit filed against EPA 

by E MA and a num ber of i ndividual e ngine m anufacturers. T he s uit c hallenged, a mong ot her 

things, c ertain s upplemental e mission r equirements r eferred t o as “ not-to-exceed” ( NTE) 

standards. On June 3, 2 003, the parties finalized a settlement of their disputes pertaining to the 

NTE s tandards. T he pa rties a greed upon a d etailed out line f or a f uture regulation t hat w ould 

require a manufacturer-run heavy-duty in-use NTE testing (“HDIUT”) program for diesel-fueled 

engines and vehicles. One section of the outline stated: 

“The N TE T hreshold w ill be t he N TE s tandard, i ncluding t he m argins bui lt i nto t he e xisting 

regulations, plus additional margin to account for in-use measurement accuracy. This additional 

margin shall be determined by the measurement processes and methodologies to be developed 

and a pproved b y E PA/CARB/EMA. T his m argin w ill be s tructured t o encourage i nstrument 

manufacturers to develop more and more accurate instruments in the future.” 

Given the foregoing, the work to be completed under this test plan is a vital component to the 

fulfillment of the settlement agreement, and it is vital to the successful implementation of a fullyenforceable H DIUT p rogram. Because of thi s significance, it is c ritically impor tant tha t the 

work de tailed i n t his t est pl an be c arried out i n a s t horough, c areful a nd t imely a m anner a s 

possible. 
REPORT 03.14936.12 A-2 

TABLE OF CONTENTS

Executive Summary ........................................................................................................ A-1

Table of Contents ............................................................................................................ A-2

List of Figures ................................................................................................................. A-4

List of Tables .................................................................................................................. A-4

1 Introduction ............................................................................................................. A-5

2 Monte Carlo Error Model and Measurement Allowance ....................................... A-7

2.1 Objective ......................................................................................................... A-7

2.2 Background ..................................................................................................... A-7

2.3 Methods and Materials .................................................................................. A-13

2.4 Simulation Procedure .................................................................................... A-13

2.4.1 Construction of the Error Surface ......................................................... A-15

2.5 Model Considerations ................................................................................... A-20

2.5.1 Convergence ......................................................................................... A-20

2.6 Simulation Output ......................................................................................... A-20

2.6.1 Sensitivity Variation Effect................................................................... A-20

2.6.2 Sensitivity Bias Effect........................................................................... A-21

3 Engine Dynamometer Laboratory Tests ............................................................... A-21

3.1 Preliminary Audits ........................................................................................ A-21

3.1.1 Objective ............................................................................................... A-21

3.1.2 Background ........................................................................................... A-21

3.1.3 On-site meeting to establish 1065 compliance requirements ............... A-22

3.1.4 Methods and Materials .......................................................................... A-22

3.1.5 Data Analysis ........................................................................................ A-22

3.1.6 PEMS Manufacturer PM PEMS Commissioning ................................. A-22

3.2 Bias and Precision Errors under steady state engine operation .................... A-23

3.2.1 Objective ............................................................................................... A-23

3.2.2 Background ........................................................................................... A-24

3.2.3 Methods and Materials .......................................................................... A-24

3.2.4 Data Analysis ........................................................................................ A-25

3.3 Precision Errors under transient engine operation (dynamic response)........ A-26

3.3.1 Objective ............................................................................................... A-26

3.3.2 Background ........................................................................................... A-26

3.3.3 Methods and Materials .......................................................................... A-26

3.3.4 Data Analysis ........................................................................................ A-27

3.4 ECM Torque and BSFC ................................................................................ A-30

3.4.1 Objective ............................................................................................... A-30

3.4.2 Data Analysis ........................................................................................ A-30

4 Environmental Chamber ....................................................................................... A-30

4.1 Data Analysis for Environmental Tests ........................................................ A-31

4.2 PM Generator Commissioning ..................................................................... A-32

4.3 Baseline ......................................................................................................... A-32

4.3.1 Objective ............................................................................................... A-32

4.3.2 Background ........................................................................................... A-32

4.3.3 Methods and Materials .......................................................................... A-32

4.3.4 Data Analysis ........................................................................................ A-33

4.4 Electromagnetic Radiation ............................................................................ A-33
REPORT 03.14936.12 A-3 

4.4.1 Objective ............................................................................................... A-33

4.4.2 Methods and Materials .......................................................................... A-33

4.5 Atmospheric Pressure ................................................................................... A-33

4.5.1 Objective ............................................................................................... A-33

4.5.2 Background ........................................................................................... A-33

4.5.3 Methods and Materials .......................................................................... A-34

4.5.4 Data Analysis ........................................................................................ A-36

4.6 Ambient Temperature and Humidity ............................................................ A-36

4.6.1 Objective ............................................................................................... A-36

4.6.2 Background ........................................................................................... A-36

4.6.3 Methods and Materials .......................................................................... A-37

4.6.4 Data Analysis ........................................................................................ A-39

4.7 Orientation and Vibration ............................................................................. A-39

4.7.1 Objective ............................................................................................... A-39

5 SwRI CVS and CE-CERT Trailer Correlation ..................................................... A-39

5.1 Method and Materials ................................................................................... A-39

6 Model Validation and Measurement Allowance Determination .......................... A-40

6.1 Model validation ........................................................................................... A-41

6.1.1 Objective ............................................................................................... A-41

6.2 Measurement Allowance Determination ...................................................... A-43

6.2.1 Objective ............................................................................................... A-43

6.2.2 Background ........................................................................................... A-44

6.2.3 Methods and Materials .......................................................................... A-44

6.2.4 Data Analysis ........................................................................................ A-44

7 Time and Cost ....................................................................................................... A-44

7.1 Timeline ........................................................................................................ A-44

7.2 Cost ............................................................................................................... A-44

8 Abbreviations used in Brake Specific Equations .................................................. A-45
REPORT 03.14936.12 A-4 

LIST OF FIGURES

FIGURE 1. PROBABILITY DENSITY FUNCTIONS FOR SAMPLING ERROR SURFACES..................................... A-8

FIGURE 2. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR METHOD 1............................................... A-9

FIGURE 3. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR METHOD 2............................................. A-10

FIGURE 4. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR METHOD 3............................................. A-11

FIGURE 5. ERROR SURFACE: PEMS VS. LAB.............................................................................................. A-14

FIGURE 6. ERROR SURFACE: (PEMS-LAB) VS. LAB.................................................................................... A-14

FIGURE 7. ERROR SURFACE: FINAL VERSION ........................................................................................... A-15

FIGURE 8. OVERVIEW OF MONTE CARLO SIMULATION ............................................................................ A-18

FIGURE 9. EXAMPLE OF A NTE CYCLE ........................................................................................................ A-29

FIGURE 10. PRESSURE HISTOGRAM........................................................................................................... A-34

FIGURE 11. PRESSURE-TIME ENVIRONMENTAL TEST CYCLE ..................................................................... A-35

FIGURE 12. TEMPERATURE HISTOGRAM................................................................................................... A-37

FIGURE 13. TIME SERIES CHART OF AMBIENT TEMPERATURE TEST ......................................................... A-38

LIST OF TABLES

TABLE 1. ALLOWED MODIFICATIONS .......................................................................................................... A-5

TABLE 2. EXAMPLE OF SELECTION OF MEASUREMENT ALLOWANCE AT 0.02 G/HP-HR NTE THRESHOLDA-12

TABLE 3. ERROR SURFACES FOR THE BSPM SIMULATION......................................................................... A-19

TABLE 4. ENGINE, EXHAUST CONFIGURATION, AND STEADY-STATE MODES ........................................... A-23

TABLE 5. EXAMPLE OF SS ERROR SURFACE ............................................................................................... A-25

TABLE 6. NTE TRANSIENT CYCLE.............................................................................................................. A-278 

TABLE 7. DYNAMIC RESPONSE INTER-NTE EVENTS ................................................................................... A-29

TABLE 8. CONCENTRATION AND DILUTION RATIO SCHEDULE WITH PM GENERATOR............................. A-31

TABLE 9. ATMOSPHERIC PRESSURE TEST SEQUENCE ................................................................................ A-35

TABLE 10. AMBIENT TEMPERATURE TEST SEQUENCE............................................................................... A-38

TABLE 11. PROJECTED PM-PEMS TIMELINE............................................................................................... A-44

TABLE 12. PROJECTED COST ESTIMATE ..................................................................................................... A-45
REPORT 03.14936.12 A-5 

1 INTRODUCTION

This te st pl an will e stablish a P EMS me asurement a llowances for P M, as r egulated b y the 

manufacturer-run on-highway heavy-duty diesel engine in-use test program. T he measurement 

allowance will be established using various laboratory facilities and PEMS. The measurement 

allowance will be established in units of brake-specific emissions (g/hp-hr), and it will be added 

to t he f inal N TE P M s tandard, after a ll t he ot her a dditive and multiplicative a llowances ha ve 

been applied. This test plan will establish the PM measurement allowance.

The PEMS used in this test plan must be standard in-production makes and models that are for 

sale as commercially available PEMS. In addition, PEMS and any support equipment must pass 

a “red-face” test with respect to being consistent with acceptable practices for in-use testing. For 

example, t he e quipment m ust m eet all s afety and t ransportation regulations f or us e on -board 

heavy-duty vehicles. 

Even though the PEMS cannot be “prototypes” nor their software “beta” versions, the steering 

committee has already agreed that after delivery of PEMS to the contractor, there may be a few 

circumstances i n w hich PEMS m odifications m ight be a llowed, but t hese m odifications mus t 

meet certain deadlines, plus they are subject to approval by the steering committee. A lso, any 

implementation of such approved modifications will not be allowed to delay the test plan, unless 

the steering committee specifically approves such a delay. Table 1 summarizes these allowable 

modifications and their respective deadlines: 

TABLE 1. ALLOWED MODIFICATIONS

Allowed Modifications Before start of...

Steering committee approved hardware and software modifications 

that af fect em issions r esults; inc luding but not limited to fittings, 

components, calibrations, compensation algorithms, sampling rates, 

recording rates, etc.

Steady-State 

Testing

Steering c ommittee a pproved ha rdware m odifications f or D OT 

approval or any other safety requirement approval 

Environmental 

Chamber Testing

Delivery of any environmental / weather enclosure to contractor Environmental 

Chamber Testing

Post-processing software to determine NTE results Model Validation

DOT approval and documentation Model Validation

Steering committee appr oved hardware or s oftware t hat i mproves 

the contractor’s efficiency to conduct testing and data reduction Always Allowed

The s teering com mittee appr oved three di fferent P EMS t hat i ncludes t he A VL M icro-Soot 

Sensor ( MSS), t he H oriba Transient P articulate Matter ( TRPM), and t he S ensors P roportional 

Particulate M atter D iluter ( PPMD). H owever, because of t he different m easurement 

technologies employed by each of these systems, the three different PEMS hold slightly different 

status w ith respect t o de termining t he P M m easurement allowance. Because i nertial 

microbalances a re already approved for PEMS a pplications in 40 C FR Part 1065, t he Sensors 

PPMD will be one of the PEMS used to determine the measurement allowance. A nd because 

EPA’s PM standard is based upon a gravimetric filter analysis, the Horiba TRPM will also be 

used to determine t he measurement al lowance. T he l owest m easurement al lowance v alue 
REPORT 03.14936.12 A-6 

between the two will be selected as the final measurement allowance for PM. If that value does 

not validate, then the lowest validated value will be chosen. If the lowest validated value chosen 

is w ithin 0.0075 g /hp-hr f rom t he l owest non -validated va lue, t hen t he l owest va lidated va lue 

will be the measurement allowance. Otherwise, the MASC will spend up to a $100,000 to figure 

out a resolution to the problem by generating more data or changing the way the validation was 

performed. If that does not lead to a resolution, then Executive Management of EMA and EPA 

will have to settle the issue.

Note tha t at the c onclusion of s uccessful te sting o f the H oriba s ystem in this me asurement 

allowance program, EPA intends to approve the Horiba system as an alternative for use, or EPA 

may elect to amend 40 CFR Parts 86 and/or 1065 to allow the use of the Horiba TRPM or other 

PEMS t hat ope rate upon similar m easurement pr inciples. B ecause t he A VL s ystem m easures 

only the soot component of PM, the measurement allowance will not be determined using the 

AVL r esults, unl ess bot h t he S ensors and H oriba s ystems f ail to complete the me asurement 

allowance program. Note that the steering committee may determine at the conclusion of the 

program that the AVL MSS is a viable alternative for demonstrating compliance. Under such a 

circumstance EPA may amend the Heavy-Duty In-Use regulation to allow for its use.

This test plan describes a computer model, a series of experiments that are used to calibrate the 

model, and another series of experiments that are used to validate the calibrated model. 

The te st pl an first describes t he c omputer m odel. T he c omputer m odel s tatistically combines 

many sources of PEMS and lab error, which are nearly impossible to capture simultaneously in a 

single te st. T he mode l w ill us e s tatistics to apply the e rrors in a way that s imulates actual 

running o f a P EMS i n-use. T he m odel w ill a lso c onsider onl y t he po rtion of e rror t hat i s 

attributable to PEMS, and it will subtract the error that is already tolerated in an emissions lab 

today. The model will also calculate and validate results according to 40 CFR Part 1065. 

The test plan then describes the series of experiments. These tests will characterize the m any 

sources of PEMS and lab error so that the specific nature of the errors can be programmed into 

the computer model. The nature of the error has to do with the way PEMS and the lab react to 

certain conditions. For example, under varying environmental conditions such as temperature or 

vibration, a PEMS might exhibit signal drift, or it may record noise that is not a part of the true 

emissions. 

Next, t he e xperimental r esults w ill be e ntered i nto t he c omputer m odel, and t he m easurement 

allowances are calculated by the model. The m odel uses a “reference" PEMS data set, which 

will ha ve m any “reference N TE ev ents.” T he model s tatistically applies a ll the e rrors to the 

reference data set, calculates results, and saves the results. Then the model will be run with all 

errors set to zero to calculate the ideal results of the reference data set. Each difference between 

a reference NTE event’s result with errors and its respective ideal result will be a brake-specific 

difference that is recorded for later use. Then the process repeats using the same reference data 

set, t o w hich ne w, s tatistically s elected e rrors are a pplied, and t hus another unique s et of 

differences is calculated. As the model continues to iterate and generate more and more results, 

patterns are expected to appear in the output data. These patterns should be the distributions of 

differences, ba sed upon the er ror t hat w as s tatistically and repeatedly ap plied to the r eference 

data set. M any difference distributions will be determined: for each reference NTE event, for 

each of the two brake-specific calculation methods (three in case of the AVL system only), and 
REPORT 03.14936.12 A-7 

for each PEMS. It has been agreed that the 95th percentile values of these distributions will be 

taken as reasonable “worst case” results for each reference NTE event. Details on how all these 

distributions will be reduced to determine the PM measurement allowance is given in the “Error 

Model” section of this test plan. 

Because the calculation based on Method 2 and Method 3 require gas-based fuel flow calculation 

based on t he measurement of CO2, CO, and NMHC, a decision w as made to use the gaseous 

PEMS data for this purpose, without the need to perform gaseous measurement during the PMPEMS program. 

Finally, the te st pl an describes how the c omputer mode l w ill be v alidated against r eal-world 

over-the-road i n-use P EMS ope ration a s w ell a s a dditional l ab t esting. For t he ove r-the-road 

testing, PEMS emissions measurements will be conducted, while at the same time a reference 

laboratory will be towed along to measure the same emissions. For the lab testing, an attempt 

will be made to simulate real-world engine operation to “replay” an over-the-road test in the lab. 

Data f rom t hese f inal experiments w ill be us ed t o va lidate t he m odel, w hich m ust be done i n 

order to gain sufficient confidence that the model did not establish unreasonable measurement 

allowances.

The following sections of this test plan are written as instructions to the contractor or contractors 

who will complete the test plan.

2 MONTE CARLO ERROR MODEL AND MEASUREMENT ALLOWANCE

2.1 OBJECTIVE

Use Monte Carlo (e.g. random sampling) techniques in an error model to simulate the combined 

effects of al l t he a greed-upon s ources of P EMS e rror i ncremental t o l ab e rror. C reate e rror 

“surfaces” for the Monte Carlo simulation to sample, based upon results from the experiments 

described in Sections 3 and 4. Exercise the model over a wide range of NTE events, based on a 

single, reference data set of at least 150 but no more than 200 unique NTE events. Determine the 

pollutant-specific brake-specific additive measurement allowance for PM.

2.2 BACKGROUND

The error model uses Monte Carlo techniques to sample error values from “error surfaces” that 

are generated from t he results of each of t he e xperiments de scribed in Section 3 on engine 

dynamometer laboratory tests and Section 4 on e nvironmental chamber tests. The lab test error 

surfaces cover the domain of error versus the magnitude of the signal to which the error is to be 

applied (i.e. 1st to 99th percentile error vs. concentration, flow, torque, etc.). T his is illustrated 

later in this section. The e nvironmental te st e rror surfaces f or s hock & vi bration a nd 

electromagnetic & r adio f requency i nterference (EMI/RFI) cover t he s ame dom ain as t he l ab 

tests. The environmental test error surfaces for pressure and temperature are characteristically 

different because they cover the domain of environmental test cycle time versus the magnitude of 

the s ignal t o which the error i s t o be applied (i.e. error at a s elected t ime vs . concentration). 

Details on how e ach s urface i s generated are given in each of t he r espective s ections. T hese 

surfaces w ill al ready b e adj usted to represent P EMS er ror i ncremental t o lab error; t herefore, 

these surfaces are sampled directly by the model.
REPORT 03.14936.12 A-8 

The error model will use two different probability density functions (PDFs) as shown in Figure 1 

to sample t he e rror s urfaces, de pending upon which e xperiment t he s urface r epresents. T o 

sample error surfaces that are generated from all the laboratory test results (Section 3), and the 

environmental t est r esults f or s hock & vi bration ( Section 4) , t he m odel w ill u se a t runcated 

normal PDF because these tests are designed to evenly cover the full, but finite, range of engine 

operation and ambient conditions. To sample error surfaces that are generated from the pressure 

and temperature e nvironmental te st r esults ( Section 4) , t he m odel w ill us e a uni form P DF 

because t hese t ests are al ready d esigned to cover t he t ypical range and frequency of t he 

respective conditions. 

 

 

FIGURE 1. PROBABILITY DENSITY FUNCTIONS FOR SAMPLING ERROR 

SURFACES

The random values that are obtained from both distributions are labeled i c in Figure 1 and range 

from -1 to 1. Note that for the pressure and temperature environmental tests, a uniform PDF 

will be used to sample test time, from which the nearest (in time) calculated errors are used. The 

errors from the other tests will be aligned with the truncated normal PDF such that each of the 

50th percentile va lues at each of t he t ested signal m agnitudes i s cent ered at t he m edian of t he 

PDF (ic = 0), and the 1st and 99th percentile error values at each of the tested signal magnitudes 

will be a ligned with the e xtreme ne gative ( ic = -1) a nd pos itive ( ic = +1) ed ges o f t he P DF, 

respectively.

Each error surface will be sampled along its i c axis (y-axis) once per reference NTE event trial, 

and it will be sampled along its parameter value axis (x-axis, e.g., concentration (only for AVL 

MSS), flow, torque, etc...) once per second, within a given reference NTE event trial. An error 

will be de termined for a given second and parameter al ong t he error ax is ( z-axis) a t the 

intersection of an ic value and a parameter value.

Probability Density Functions for Sampling Error Surfaces Once Per NTE Event 

-1.00

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

5 4 3 2 1 0

Relative Probability

i c

Lab Tests, Normal, SD=0.60795, truncate @ -1 & 1

Environmental Tests, Uniform

Note: A non-truncated normal 

distribution with SD=0.60795 has P 

values of 0.01 and 0.99 at i c =-1 and 

ic=+1, respectively.
REPORT 03.14936.12 A-9 

To e nsure t hat t he m agnitudes of t he error s urfaces are a ppropriate, each da ta poi nt us ed t o 

generate the surfaces will be a mean or a weighted mean of 30 seconds of sampling.

Interpolation will be p erformed b y first line arly int erpolating e rror values at ea ch tested 

magnitude al ong t he s elected line pe rpendicular t o the i c axis. T hen f rom t hat l ine of e rrors, 

individual error values will be linearly interpolated at each second-by-second signal magnitude 

of the given NTE event in the reference data set. 

The r eference d ata s et t o which all er rors w ill be appl ied will be a l arge d ata s et of engine 

operation over a wide range of NTE events. This reference data set will be initially generated 

from col lections of real-world PEMS data sets. The reference data s et s hould contain at l east 

150 but no m ore t han 2 00 uni que N TE e vents. P arameters i n t he r eference da ta s et m ay be 

scaled in order t o exercise t he m odel t hrough a m ore appr opriate r ange of p arameters ( i.e. 

concentrations, f lows, ambient c onditions, e tc.). If t he pa rameters a re scaled, c are s hould be 

taken to maintain the dynamic characteristics of the reference data set.

After the errors are applied, NTE brake-specific PM emissions results are calculated, using each 

of the three agreed-upon NTE calculation methods. The three different brake-specific emission 

calculation methods for PM referred to in this test plan are i) Torque-Speed method, ii) BSFC 

method, a nd i ii) E CM-Fuel S pecific m ethod, and these ar e i llustrated in Figure 2, 3, a nd 4 , 

respectively.

For all PM PEMS:

m PM is a flow weighted particulate matter exhaust concentration in g/mol 

( ) ( ) ( ) ∑

∑

=

=





 



 ⋅ ∆





 



  ∆ 

 



 



 





=⋅ N

i

i i

N

i

PM i

PM rpmSpeed mNT t

t

s

mol

n

mol

g m

ge kW hr

1

1

3600*1000*60

*14159.3*2**

* *

/



Where for AVL:

m PM

1

1

*

*

N

i i

i

PM N

i

i

g mol mPM n t

g mol s

m

mol mol

n t

s

=

=

        ∗ ∆            =         ∆      

∑

∑





is computed numerically as follows, 

FIGURE 2. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR METHOD 1
REPORT 03.14936.12 A-10 

For all PM PEMS:

m PM is a flow weighted particulate matter exhaust concentration in g/mol 

 

( )

( ) ( ( ) ( ))

( ) ( )

1

6 2

2

* *

/

* *10 % % *10 *

*

* *2*3.14159

60*1000*3600

i

N

PM i

i

PM

ii i

C

fuel

fueli

i i

g mol m nt

mol s e g kW hr

mol n xTHC ppm xCO xCO t

M s

w g m

s

Speed rpm T N m

=

− −

      ∆       ⋅ =  

 

      ++ ∆    

         

⋅

     

∑ 



 1

N

i=

∑

Where for AVL:

m PM

1

1

*

*

N

i i

i

PM N

i

i

g mol mPM n t

g mol s

m

mol mol

n t

s

=

=

        ∗ ∆            =         ∆      

∑

∑





is computed numerically as follows, 

FIGURE 3. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR METHOD 2
REPORT 03.14936.12 A-11 

For AVL Only:

( )

( ) ( ( ) ( ))

( ) ( )

6 2

1 2

1

/

* * *

*10 % % *10

* *2*3.14159*

60*1000*3600

i

PM

N ifue l

fuel

PM

C i i i

N

i i

i

e g kW hr

g m

g w s m t

mol M xTHC ppm xCO xCO

Speed rpm T N m t

− − =

=

⋅ =

              ∆   + +

 

  ⋅ ∆

   

∑

∑



Where:

( ( ))

( ) ( ( ) ( ))

( ) ( ( ) ( ))

6 2

1 2

6 2

1 2

/ *

* *

*10 % % *10

* *

*10 % % *10

i

i

N i ifue l

fuel

C i i i

PM

N ifue l

fuel

C i i i

g mPM g mol m w s t

M xTHC ppm xCO xCO

g m

mol g m

w s t

M xTHC ppm xCO xCO

− − =

− − =

          ∆

+ +

      =             ∆

+ +

 

∑

∑





FIGURE 4. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR METHOD 3

Next, the NTE events are calculated by each of the three calculation methods, but with no error 

sampled or applied to the reference data set. These results are considered the “ideal” results of 

the reference NTE events. These ideal results are subtracted from each respective NTE event 

result ‘with errors’, and the difference is recorded. T hen a new set of errors are sampled and 

applied to the reference NTE event, and the NTE results ‘with errors’ are calculated again. The 

ideal results are again subtracted, and the difference is recorded. T his is repeated thousands of 

times so that the model converges upon distributions of brake-specific differences for each of the 

original NTE events in the reference data set. 

Then the 95th percentile difference value is determined for each NTE event distribution of brakespecific differences for PM for each calculation method. At this point there is one distribution of 

95th percentile differences for PM, where all the NTE events are pooled by the PM emissions for 

each of t he t hree di fferent c alculation m ethods. E ach of t he 95 th percentile di stributions 

represents a range of possible measurement allowance values.

From e ach of t hese t hree di stributions of p ossible m easurement a llowance v alues, one 

measurement allowance per distribution must be determined. First the correlation between 95th

percentile differences versus the ideal PM emission is tested. For each calculation method, if a 

least squares linear regression of 95th percentile differences versus ideal PM emissions has an r2 

(squared correlation coefficient) > 0.85 and an SEE (standard error of the estimate or root-mean-
REPORT 03.14936.12 A-12 

squared-error) < 5 % of the median ideal PM emission, then that linear regression equation will 

be us ed to determine t he m easurement allowance f or that calculation method at the f ollowing 

NTE threshold: 

PM = 0.02 g/hp-hr and 0.03 g/hp-hr 

In cases where ex trapolation is r equired to determine t he m easurement allowance at t he N TE 

threshold, t he m easurement a llowance w ill be de termined u sing t he l inear r egression, but 

evaluated at the ideal PM emission that is closest to the NTE threshold, not extrapolated to the 

NTE t hreshold i tself. If t he l inear r egression does not pa ss t he aforementioned r 2 and SEE 

criteria, then the m edian value of the 95 th percentile di fferences i s us ed as t he s ingle 

measurement allowance for that calculation method. 

Next, the calculation method is selected. T he above procedure will provide three measurement 

allowances, where applicable, one for each of the three different calculation methods. To make 

them com parable, the t hree m easurement al lowance va lues w ill be n ormalized by t he P M 

threshold and expressed as a percent. Also, if any measurement allowance is determined to have 

a va lue l ess t han zero, t hen that m easurement al lowance w ill be s et equa l t o zero. T he 

calculation m ethod w ith t he m inimum nor malized P M va lue w ill be c hosen a nd t he 

corresponding normalized PM value will be selected as the best measurement allowance for PM, 

assuming it va lidates. If it doe s not va lidate, then the mini mum va lue that va lidates w ill be 

chosen as long as it is within 0.0075 g/hp-hr from the minimum value that did not validate. If the 

difference b etween the minimum va lue tha t va lidates a nd the mini mum va lue tha t di d not 

validate i s greater t han 0.0075 g/hp-hr, additional i nvestigation w ith up t o a $100,000 w ill be 

spent in order to understand why the minimum value chosen did not validate. If the problem is 

not r esolved a fter s pending t he $100,000, t hen t he m atter w ill be r eferred to executive 

management of EPA and EMA to decide on the PM measurement allowance.

 

Table 2 below i llustrates t he s election of t he c alculation m ethod. T he example i s ba sed on a 

hypothetical s et of nor malized PM me asurements f or the th ree calculation m ethods. The 

minimum of these normalized allowances is used to select the best method (highlighted in blue). 

In this hypothetical case, the BSFC method would be selected. 

TABLE 2. EXAMPLE OF SELECTION OF MEASUREMENT ALLOWANCE AT 0.02 

G/HP-HR NTE THRESHOLD

 Allowance at Respective NTE Threshold (%)

Calc. Method ==> Torque-Speed BSFC ECM fuel specific

BSPM 38 % 18 % N/A

Selected Method==> BSFC Method

Therefore, 18% w ould be s elected as t he be st m easurement al lowance f or P M, assuming i t

validates. Otherwise, the 38 % will be chosen if it validates. T hus, the additive brake-specific 

measurement allowance would be:

PM = 18 % * 0.02 g/hp-hr = 0.0036 g/hp-hr, if it validates, and if not, then: 
REPORT 03.14936.12 A-13 

 PM = 38 % *0.02 g/hp-hr = 0.0076 g/hp-hr, if it validates, and if not, then: 

spend up to a $100,000 to figure out why it did not validate in the first place, and then apply the 

above strategy again, assuming the value now validates. If not, then EPA and EMA executive 

management will decide on the PM measurement allowance value. 

This PM value would be the value added to the actual brake-specific NTE threshold for a given 

engine, based on actual family emissions limit, mileage, model year, etc.

2.3 METHODS AND MATERIALS 

Exercise the model using three different calculation methods: a) Torque-Speed method, b) BSFC 

method, and c) ECM-Fuel Specific method (only for AVL MSS). Determine which calculation 

method is the most accurate, and use it to estimate the measurement allowance. Each calculation 

method is described in Figured 2, 3, and 4. 

Prepare an Excel spreadsheet model for use with the Crystal Ball Monte Carlo software for error 

analysis o f br ake s pecific e missions, B SE, a s ou tlined i n s ection 2.4. C hanges t o t he m odel 

specifications m ay b e requested as a greed up on by t he S teering C ommittee. Prepare t he 

spreadsheet in a modular structure following the specified model outline, and make provisions 

for the identified calculation modules. Additionally, clearly identify and easily locate input cells 

to the model to facilitate any revisions that may become necessary for users who want to exercise 

the model with other Monte Carlo add-ins such as @Risk or the newest versions of Crystal Ball. 

Test the spreadsheet with controlled test cases of simplified input distributions with the Crystal 

Ball add-in to confirm correct model implementation in accordance with this test plan. Run at 

least one typical analysis as an additional confirmation.

Deliver t he electronic s preadsheet and a br ief r eport de scribing t he model, pr esenting t he t est 

cases, and describing pertinent information including the Crystal Ball version number, the Excel 

version number, the operating system and the computer. U se standard spreadsheet calculations 

so that no serious di fficulties w ill be anticipated r egarding a pplication i n ot her s preadsheet 

versions. Use Crystal Ball Version 7 or higher, and confirm test cases using Excel 2003. 

Control revisions of the spreadsheet model using descriptive file names. Extensive revisions or 

testing with other software v ersions be yond that initially proposed m ay be re-proposed b y the 

Steering Committee if and when a need for such additional work is identified. 

2.4 SIMULATION PROCEDURE

For each of the measurement errors in Section 3, create an error surface and sample it according 

to the aforementioned PDFs. Each error surface represents an additive error—or a subtractive 

error if the sign is negative—relative to the reference value to which it is applied. F igure 5, 

Figure 6, and Figure 7 serve as a hypothetical PM example of how these error surfaces should be 

created f or e very e rror. T he pl ots s hown c orrespond t o PM emissions c oncentration da ta 

representing 1 P EMS, t wo e ngines, and t hree exhaust c onfigurations e ach, w ith a ll 6 s ets of 

PEMS data pooled together. Note that separate error surfaces will be constructed for each of the 

three P EMS uni ts ( AVL, H oriba a nd S ensors). The e xample a pplies t o the e rror m odule f or 
REPORT 03.14936.12 A-14 

steady-state (SS) bias and precision PM concentration errors (Section 3.2). These figures will 

be referenced by each “Data Analysis” section for the various errors discussed in this test plan. 

FIGURE 5. ERROR SURFACE: PEMS VS. LAB

FIGURE 6. ERROR SURFACE: (PEMS-LAB) VS. LAB

Error Surface for SS PM Concentration

18

26

38

45

56

69

13

24

36

42

53

64

8

19

33

39

51

61

0

10

20

30

40

50

60

70

80

0 10 20 30 40 50 60 70 80

PM μg/mole (lab,mean)

PM μg/mole (PEMS)

diagonal 95th percentile

50th percentile (median) 5th percentile

Error Surface for SS PM Concentration

8

6

8

5

6

9

3

4

6

2

3

4

-2

-1

3

-1

1 1

-4

-2

0

2

4

6

8

10

0 10 20 30 40 50 60 70

PM μg/mole (lab,mean)

PM μg/mole (PEMS) - PM μg/mole (lab)

95th percentile 50th percentile (median) 5th percentile
REPORT 03.14936.12 A-15 

FIGURE 7. ERROR SURFACE: FINAL VERSION

Errors f rom Section 3 ( Engine D ynamometer Laboratory t ests) and S ection 4 ( Environmental 

Chamber Tests) are combined by adding all of the sampled errors once per NTE event trial. For 

example, in order to assess the errors in PM concentration for each NTE event, several modules 

will be created such that:

PM_with errors = PM _ideal + ∆(μg/mole)1 + ∆(μg/mole)2 + ∆(μg/mole)3 + ...

where,

 ∆(μg/mole)1 

∆(μg/mole)

= PM concentration errors due to steady state bias and precision errors, 

2 

∆(μg/mole)

= PM concentration errors due to ambient temperature,

3 

etc....

= PM concentration errors due to ambient pressure, 

2.4.1 Construction of the Error Surface

2.4.1.1 PEMS vs. Lab 

Acquire raw data with the PEMS at various average concentration levels as per Section 3.2. Plot 

the “P EMS” s ignals ve rsus t he c orresponding “lab” s ignals t hat w ere m easured us ing l ab 

equipment. This plot pools all bias and precision errors for one PEMS and for all data from all 

engines for all steady-state modes. Shown in Figure 5 are the 5th, 50th and 95th percentiles at the 

mean PM concentration level from the lab (note that the distribution of data at each level is not 

necessarily Gaussian). If t he 50 th

Error Surface for SS PM Concentration

10.1 6.8 8.8 7.2 11.1

3 4 6 2 3 4

-4.1 -3.1 1.8 -2.2 0.2 -0.2

-1.5

-1

-0.5

0

0.5

1

1.5

0 10 20 30 40 50 60 70

PM μg/mole (lab,mean)

ic_SS_PM

99th percentile 50th percentile (median) 1st percentile

Error Surface: z-axis = ΔSS_PM_μg/mole

6.

percentile is di fferent tha n the line of p erfect a greement 

(diagonal), the data suggests that there is a bias error between PEMS and Lab. In essence this 

graph s hows t he s tatistical di stribution m easured b y the P EMS a t each a verage c oncentration 
REPORT 03.14936.12 A-16 

level sampled. The example shows only 6 discrete PM concentration levels (ranging from 10-60 

μg/mole). However, the actual number of discrete levels will be determined by the total number 

of operating conditions actually run for all the tests of all the engines. For example, the SS PM 

testing will select 6 modes representing typical operating conditions. Thus, the actual plot for SS 

PM will likely have 36 discrete concentration levels (6 modes x 1 PEMS x 2 engines x 3 exhaust 

configurations). 

2.4.1.2 (PEMS – Lab) vs. Lab 

The plot in Figure 6 basically shows the “additive error band” measured during testing. The plot 

is created by first subtracting the “lab” PM value from the corresponding individual PEMS PM 

measurement for each test run. This difference is defined as the ‘delta’ error. Next, the “PEMS 

– Laboratory” delta errors are pooled at each average lab PM value to obtain the 95th, 50th, and 

5th percentile values, respectively, displayed in Figure 5. N otice that if lab error exceeds PEMS 

error at a given percentile, crossover o f va lues can occur. T his i s ac ceptable b ecause t he 

crossover effectively reduces PEMS error whenever lab error exceeds PEMS error.

In order to obtain estimates of the 1st and 99th percentiles for the delta errors for a given “lab” 

PM value, each side of the corresponding error distribution will be assumed to independently fit 

a normal di stribution. Because o f t he a symmetry of t he da ta, t his m ethodology w ill yield t wo 

halves of a nor mal di stribution. The m edian of e ach nor mal di stribution w ill be the m edian 

based on t he delta errors given in Figure 6. The 95th percentile delta error will form the upper 

boundary of one half of the normal distribution, and the 5th percentile delta error will form the 

lower bounda ry of t he ot her ha lf of t he nor mal di stribution. W hen e ach s ide of t he da ta 

distribution is fitted to a normal distribution using the above boundary conditions, one can then 

expand each half of the distribution from the error surface to obtain the 1st and 99th percentiles of 

the data for the given “lab” PM value.

2.4.1.3 Error Surface

This step normalizes the data in Figure 7 using what is called a “variability index (ic)”, which 

represents the random sampling by the Monte Carlo technique, in order to select a given error 

level. This variability index is allowed to vary from –1 to +1. T he likelihood of i c being any 

value between –1 through +1 is specified by the PDF assigned to ic. In the given example, i c is 

assumed t o va ry according t o a nor mal di stribution dur ing M onte C arlo calculations. T his i s 

because it is believed that the distribution of errors due to steady-state bias and precision will be 

centered about the 50th percentile of the full range of conditions measured according to Section 

3.2. The pressure and temperature environmental error modules use uniform probability density 

functions for their respective variability index. Each set of data for each lab set-point mean (i.e., 

lab reference value) in Figure 6 is normalized by aligning the 1st percentile error from the fitted 

normal distributions with ic = -1, the 50th percentile error with ic = 0, and the 99th percentile error 

from the fitted normal distribution with ic = +1. 

Error surfaces such as the one presented in Figure 7 are the input modules that the Monte Carlo 

simulation program will use during calculations of brake-specific PM emissions. For example, 

for a given NTE calculation a random ic value is chosen once per NTE event trial. Let us assume 

that the first random sample produced an ic = 0.5. Let us also assume that during this NTE event 

trial, the reference PM concentration is 10 μg/mole. In this case,
REPORT 03.14936.12 A-17 

 ∆(μg/mole)1 = (3 + 10.1) / 2 = 6.55 μg/mole.

Also, from Figure 7, for ic = 0.5, the reference PM = 10 μg/mole.

For that step in the calculation, the Monte Carlo approach will add this “delta” to the reference 

concentration value of 10 μg/mole (10 μg/mole + 6.55 μg/mole = 16.55 μg/mole) to represent 

errors in steady-state bias and precision for i c = 0.5, and reference NTE PM = 10 μg/mole. If 

during the same NTE event in the reference data set, a reference concentration of 35 μg/mole is 

read, then, 

 ∆(μg/mole)1 = ((6 + 8.8) / 2 + (2 + 6.2) / 2) / 2 = 5.75 μg/mole (from Figure 7)

Note that first the error along the i c line perpendicular to the i c axis (in this case the line along 

0.5) is linearly interpolated at each discrete concentration level. T hen those interpolated values 

are t hemselves l inearly interpolated to determine t he e rror corresponding t o each reference 

concentration in the NTE event. Note that the random selection is once per reference NTE event 

trial, but the error along that i c line is applied to every second-by-second value within the given 

reference NTE event, except for PM concentration in the case of Horiba and Sensors, where no 

second-by-second i nformation a re a vailable, b ut di fferent P M c oncentration l evels m ay be 

available for a specific NTE event. 

Now let us assume that the error in PM concentration is composed of only 3 deltas: ∆(μg/mole)1, 

∆(μg/mole)2 , and ∆(μg/mole)3 . And let us assume that for a given reference NTE event trial 

we have the following values: 

• Reference PM at one second= 30 μg/mole 

• ∆(μg/mole)1 

• ∆(μg/mole)

= 6 μg/mole

2 

• ∆(μg/mole)

= -2 μg/mole 

3 = -3 μg/mole. 

When the model calculates brake-specific emissions by each of the three calculation methods, it 

will use the following PM value, which has all of its error applied:

PM = 30 + 6 –2 – 3 = 31 μg/mole.

The appl ication of er ror at t he f irst s elected ic continues dur ing t he e ntire N TE e vent w ithout 

having to randomly sample again. In other words, i c

mPM

will not change during that random trial. 

For all of the variables except for , the errors may continue to change during an NTE event 

on a s econd-by-second ba sis i f t heir e rror s urface ha ppens t o be a f unction of l evel. F or t he 

second randomly selected i c this entire process of determining the ∆μg/mole errors is repeated. 

The s imulation will c ontinue to randomly s elected ic values f or t housands of t rials unt il 

convergence is met.

For the Horiba and Sensors generated reference NTE events, there is only one flow-weighted PM 

value for the entire NTE event. D uring the simulation for these t ypes of reference NTEs, the 

single PM value will be used in the interpolation of the corresponding PM error surfaces (i.e., 
REPORT 03.14936.12 A-18 

steady-state PM, transient PM) at all seconds of the reference NTE event. Since the PM value 

will not vary from second-to-second, the only interpolation will occur according to the ic value at 

each of the simulation trials.

The same second-by-second sampling and interpolation approach would be used for other deltas 

such a s a mbient t emp, a mbient pr essure, s hock a nd vi bration, B SFC i nterpolation, t orque, 

exhaust f low r ate, e tc. A n ove rview o f t he Monte C arlo s imulation f or P M i s de tailed i n 

Figure 8. 

FIGURE 8. OVERVIEW OF MONTE CARLO SIMULATION
REPORT 03.14936.12 A-19 

Table 3 lists the e rror s urfaces tha t w ill be c reated for us e in simulating the B SPM e rror 

differences.

TABLE 3. ERROR SURFACES FOR THE BSPM SIMULATION

Calculation 

Component Test Source Error Surface

Delta PM Engine Dyno Delta PM SS

Delta PM Engine Dyno Delta PM Transient

Delta PM Environ Delta PM Ambient Temperature

Delta PM Environ Delta PM EMI/RFI

Delta PM Environ Delta PM Atmospheric Pressure

Delta PM Environ Delta PM Vibration

Delta CO Engine Dyno Delta CO SS

Delta CO Environ Delta CO Atmospheric Pressure

Delta CO Environ Delta CO Ambient Temperature

Delta CO Engine Dyno Delta CO Time Alignment

Delta CO2 Engine Dyno Delta CO2 SS

Delta CO2 Engine Dyno Delta CO2 Transient

Delta CO2 Environ Delta CO2 Ambient Temperature

Delta NMHC Engine Dyno Delta NMHC SS

Delta NMHC Engine Dyno Delta NMHC Transient

Delta NMHC Environ Delta NMHC Atmospheric Pressure

Delta NMHC Environ Delta NMHC Ambient Temperature

Delta NMHC Environ Delta Ambient NMHC

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow SS

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow Transient

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow Pulsation

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow Swirl

Delta Exhaust Flow Environ Delta Exhaust EMI/RFI

Delta Exhaust Flow Environ Delta Exhaust Temperature

Delta Exhaust Flow Environ Delta Exhaust Pressure

Delta Torque Engine Dyno Delta Dynamic Torque

Delta Torque Engine Dyno Delta Torque DOE Testing

Delta Torque Engine Dyno Delta Torque Warm-up

Delta Torque Engine Dyno Delta Torque Humidity/Fuel

Delta Torque Engine Dyno Delta Torque Interpolation

Delta Torque Engine Manuf Delta Torque Engine Manuf

Delta Speed Engine Dyno Delta Dynamic Speed

Delta Fuel Rate Engine Dyno Delta Dynamic Fuel Rate
REPORT 03.14936.12 A-20 

2.5 MODEL CONSIDERATIONS

2.5.1 Convergence

The ma in goal of the c onvergence criteria is to define how ma ny s imulation trials a t a given 

reference NTE event are required to estimate the 95th percentile BSPM emission differences with 

a given precision. T he convergence method to be used is based on a nonparametric statistical 

technique3 which de fines a 90% confidence i nterval f or t he 95 th percentile of t he BSPM 

emissions di fferences f or an individual r eference N TE s imulation. I f t he w idth of t he 90% 

confidence interval is less than 1% of the BSPM emissions threshold, then convergence is met. 

The following steps define the convergence method: 

1. Run the Monte Carlo simulation for N trials for a single reference NTE event.

2. Order the BSPM emissions differences from smallest to largest.

3. Identify the trial number at the lower end of the 90% confidence interval 

nlower = N − *05.0*95.0645.1*95.0 N

4. Identify the trial number at the upper end of the 90% confidence interval

nupper = N + *05.0*95.0645.1*95.0 N

5. Compute (BSPM difference value at nupper) – (BSPM difference value at nlower

6. If the result in (5) < 1% of the BSPM emissions NTE threshold (0.02 g/hp-hr) then 

convergence is met.

).

2.6 SIMULATION OUTPUT

It is i mportant t o unde rstand a nd i dentify w hat e rror s urfaces ha ve t he m ost influence ( i.e., 

sensitivity) on t he B SPM e missions ‘ with e rrors’ a nd, t hus, t he resulting B S e missions 

differences. Contributions to sensitivity can be attributable to changes in variance and/or bias.

2.6.1 Sensitivity Variation Effect

During the Monte Carlo simulation for each reference NTE event, sensitivity charts produced by 

Crystal B all w ill be g enerated and stored in output R EPORT f iles. Crystal B all c alculates 

sensitivity b y computing t he r ank c orrelation coefficient be tween every assumption ( error 

surface) and forecast value (delta BS emissions) while the simulation is running. Positive rank 

correlations i ndicate t hat an increase i n the assumption is as sociated with an increase i n t he 

forecast. The larger the absolute value of the rank correlation the stronger the relationship. 

Sensitivity charts in Crystal Ball provide a means to determine how the variances of the error 

surfaces affect the variance in the forecast values. Hence, the sensitivity charts developed during 

a simulation are displayed as “Contribution to Variance” charts which are calculated by squaring 

the r ank c orrelation coefficients for all a ssumptions us ed i n a pa rticular f orecast a nd t hen 

normalizing the m to 100%. T he assumption ( error s urface) w ith t he hi ghest c ontribution t o 

variance (in absolute value of the percent) is listed first in the sensitivity chart.

Simulation results f rom a ll r eference NTE e vents w ill pr oduce s ensitivity values f or the 95 th

percentile delta PM emissions by all three calculation methods.
REPORT 03.14936.12 A-21 

2.6.2 Sensitivity Bias Effect

Another type of sensitivity to be examined in this study is concerned with the effects of potential 

“bias” in error surfaces and their effects on the forecast values. In order to study these effects a 

new error surface assumption will be added to the simulation model for each of the original error 

surfaces.

This a ssumption w ill be s ampled a s a di screte binary distribution ( i.e., on or of f) during t he 

simulation. For each trial of the simulation, the original error surfaces and ‘on/off’ error surfaces 

will be s ampled a ccording t o t heir de fined s ample di stribution. I f t he ‘ on/off’ e rror s urface 

produces a n ‘off’ c ondition, t he de lta e missions f rom t hat pa rticular error s urface w ill not be 

added to the BSPM emissions computations for the BSPM emissions ‘with errors’. Similarly, if 

the ‘ on/off’ error s urface pr oduces a n ‘ on’ c ondition, t he de lta e missions f rom t hat pa rticular 

error surface will be added to the BSPM emissions calculations.

During e very t rial of t he s imulation, t he exclusions due t o t he ‘ off’ c onditions w ill r esult i n 

various combinations of the error surface delta emissions being added to the BSPM emissions 

‘with e rrors’ computations. O ver t he c ourse of a s imulation with thousands of tr ials, the 

sensitivity of a pa rticular error either ‘on’ or ‘off’ will be assessed by examining the change in 

the forecast delta emission. Therefore, in a single Monte Carlo simulation of a r eference NTE 

event sensitivities due to variance and/or bias will be explored. 

3 ENGINE DYNAMOMETER LABORATORY TESTS

Utilize e ngine d ynamometer la boratory te sting t o establish the di fference be tween PM P EMS 

and PM based on laboratory measurement in accordance with Part 1065. Also establish how well 

ECM parameters can be used to estimate torque and BSFC.

First, however, audit all the PEMS and lab equipment to ensure that they are operating properly, 

according t o 40 C FR P art 1065, S ubpart D . N ext, c onduct s teady-state engine d ynamometer 

tests to establish PEMS s teady-state bi as a nd pr ecision r elative t o t he l ab. T hen, c onduct 

transient e ngine d ynamometer te sting to determine P EMS tr ansient pr ecision by r epeating 

transient NTE events. Finally, compare ECM derived torque and BSFC to laboratory measured 

torque and BSFC. 

3.1 PRELIMINARY AUDITS

3.1.1 Objective

Conduct 40 CFR Part 1065, Subpart D audits of all engine dynamometer laboratory systems and 

all PEMS.

3.1.2 Background 

Because the overall purpose of this entire test plan is to establish measurement allowance that 

account for the incremental difference in the performance of PEMS versus engine dynamometer 

laboratory s ystems, the f irst task is to audit all of the measurement s ystems to ensure that the 
REPORT 03.14936.12 A-22 

specific systems used for testing meet EPA’s minimum performance requirements. T he audits 

also help to minimize bias errors between PEMS and lab s ystems measurements. However, in 

case a specific PM-PEMS does not meet the specifics of Part 1065 requirement, the MASC will 

decide on how to move forward by perhaps allowing some flexibility in passing Part 1065 audit, 

in situations where it mi ght be needed, especially if the performance of a system is within the 

expectation of the manufacturer.

3.1.3 On-site meeting to establish 1065 compliance requirements

In order to clarify what are all the requirements expected from the lab-grade instrumentation and 

PEMS equipment, with respect to 1065 compliance, a meeting will be held between the test plan 

steering c ommittee a nd the c ontractor at the c ontractor s ite to provide t he c ontractor w ith 

guidance regarding which specific sections of Part 1065 S ubpart D are required and which are 

optional. In case Part 10 65 requirement is demonstrated to be too stringent or impractical, the 

contractor may seek approval from the MASC to lessen the stringency of Part 1065 in relation to 

the PEMS.

3.1.4 Methods and Materials 

Use the methods and materials described in 40 CFR Part 1065, Subpart D to conduct audits of all 

lab a nd P EMS m easurement s ystems. E ven i f l ab s ystems a nd P EMS pa ss initial S ubpart D 

audits, allow lab operators and PEMS manufacturers to make on-site adjustments to improve the 

performance o f t heir s ystems pr ior t o e ngine t esting. A llow a djustments t o be ba sed on 

recalibrations w ith reference s ignals t hat a re a llowed in 40 C FR P art 1065. T he s teering 

committee may direct the contractor to calibrate or adjust the laboratory sampling system based 

on audit results. The steering committee may also suggest that a PEMS manufacturer calibrate 

or adjust one or more PEMS based on lab audits. 

3.1.5 Data Analysis 

Use t he da ta a nalyses d escribed i n C FR P art 1 065 S ubparts D , J a nd G. F or a ll s ubsequent 

testing, use onl y thos e measurement s ystems tha t pa ss the mini mum p erformance c riteria in 

Subpart D, unless a deficiency is deemed acceptable in writing b y all parties including PEMS 

manufacturers. P rovide a l ist a nd br ief de scription of a ll t he a udits c onducted f or each P EMS 

manufacturer type. EPA would likely use this list as a template for the data requirements in the 

PM portion of the HDIU testing program.

3.1.6 PEMS Manufacturer PM PEMS Commissioning

Notify PEMS manufacturers when the 1065 audits are complete and the first set of PM PEMS 

are completely ins talled in the e ngine d ynamometer te st c ell—in pr eparation f or emissions 

testing. Schedule dates and times that are prior to the start of emissions testing for each PEMS 

manufacturer to conduct a final commissioning of all their PEMS that are on site, including those 

PEMS that are not installed in the test cell. PEMS manufacturers may inspect their PEMS and 

make a ny f inal a djustments to their r espective PEMS in order f or the P EMS to meet the ir 

specifications. Allow PEMS manufacturers to inspect the installation of their PEMS in the test 

cell. If P EMS m anufacturers t ake exception to any portion of t he i nstallation or on -site 

configuration, a ttempt t o r esolve any s uch i nstallation i ssues. If s uch i ssues a re not e asily 
REPORT 03.14936.12 A-23 

resolvable, notify the steering committee, who will determine a course of action. Once PEMS 

manufacturers ha ve com pleted their commissioning, notify the s teering c ommittee. From thi s 

point any further modifications to the PEMS may only be made according to Table 1 of this test 

plan. 

3.2 BIAS AND PRECISION E RRORS UNDE R S TEADY S TATE EN GINE 

OPERATION

3.2.1 Objective

Evaluate the bias and precision using one engine and one exhaust configuration, shown in Table 

4, a nd 10 r epeats of s teady-state m odes, a nd t hree s ets of P EMS uni ts, each s et i ncluding t he 

MSS, TRPM, and PPMD. Thus, the total number of NTE steady-state points required to conduct 

the s teady-state experiments is 30. This constitutes six steady-state modes of engine operation 

(6), 10 r epeats (10), one exhaust configuration, one engine (1), and three different PEMS units 

(3), 6x10x1x1x3= 180. 

TABLE 4. ENGINE, EXHAUST CONFIGURATION, AND STEADY-STATE MODES

No. of Steady-State Modes 

for Bypass Setting 1 

(BSPM and PM 

Concentration, 

representative of PM 

threshold of 0.025 g/hp-hr 

under NTE Transient 

Operation) PM-PEMS Units Number of Repeats

07 Engine 

1 

SS1, SS2, SS3, SS4, SS5, 

SS6

Three Sets of (MSS, 

TRPM, and PPMD) 

10 per Mode per PMPEMS Set

Determine t he ( ) g ΔSS PM m mol surface plots for the error model based upon a ll data pooled. Note 

that e ach br and of P EMS w ill ha ve i ts ow n ΔSS PM m error s urface generated for us e i n bot h 

calculation methods 1 and 2. For calculation method 3, t he AVL brand PM PEMS will have a 

unique ΔSS PM m calculated according to Figure 4 of this test plan. 

Recommend six steady-state points based on the PM measurement, using the AVL MSS, of 80 

SS points of the Cummins cycle that is typically used to generate ECM torque and BSFC errors 

versus laboratory. The MASC will accept the six steady-state points or choose alternative points 

for each exhaust configuration. The objective for the MASC will be to select steady-state points 

within a given exhaust configuration that provides a nominal spread of concentrations within that 

configuration’s target brake-specific levels. Note that to achieve the brake-specific targets under 

steady-state conditions, the bypass might have to be opened further, relative to the transient NTE 

bypass settings.
REPORT 03.14936.12 A-24 

3.2.2 Background 

Testing will be conducted to capture bias and precision errors in PEMS’ emissions instruments 

versus the laboratory filter-based method. The tests will be steady-state only. 

Note: S ection 3.3 ( next s ection) w ill e valuate pr ecision e rrors ( not bi as) due to the d ynamic 

response of the PEMS instrumentation. The precision error captured during steady state testing 

(section 3.2) will have to be subtracted from the overall precision error captured in section 3.3 in 

order not t o doubl e-count t he s teady s tate precision e rrors of P EMS i nstrumentation. T his 

process is detailed in Section 3.3. 

3.2.3 Methods and Materials 

a) One model year 2007 heavy duty diesel engines, equipped with a DPF in the exhaust (Mack 

MP9) 

Use the following systems:

b) Nine PM PEMS (3 Sensors PPMD, 3 AVL MSS, 3 Horiba TRPM) 

c) One PEMS exhaust flow-meter from Sensors, Inc., and one and from Horiba, applicable to 

the engine to be tested

d) DPF with Bypass Setting 1 for SS testing, representing a threshold level of about 0.025 g/hphr under NTE transient testing

e) Measure PM via the CVS, Part 1065 Lab Method (most recent publication) 

Use the following overall guidelines:

f) Measure engine inlet airflow through use of LFE or equivalent 

g) Use a series of six steady-state modes, and set each mode time to collect a CVS filter mass of 

at least 75 microgram per mode, simultaneously with other PM-PEMS

h) Regenerate DPF system prior to each series of steady-state tests 

i) Capture ECM broadcast channels and other common diagnostic channels, as recommended 

by engine manufacturer(s), to ensure proper engine operation 

j) Do not measure gaseous species by the PEMS

k) Stabilization time = 180 seconds, with a different running time per mode to achieve a 75 

microgram or higher of PM on the CVS filter 

l) Always power off PEMS equipment at end of each day, according to PEMS manufacturer 

instructions. Re-start start-up process every day according to PEMS manufacturer 

instructions and Part 1065, Subpart J. 

m) Whenever PEMS are exchanged, swap the order of the Horiba and Sensors flowmeters, if the 

steup allows for it. 

6 point steady-state repeat-testing, evaluate bias and precision errors:

a) The MASC will select 6 SS operating conditions for repeat testing from a matrix of 80 SS 

points containing information on PM emissions using the AVL MSS 

b) Randomize the order of the six modes 

c) Repeat each six steady-state cycle two or three times, prior to DPF regeneration

d) Each test will use three PEMS (Sensors, AVL, and Horiba) at a time, to measure PM 

emissions concentration and exhaust flow rate. 

e) Expected test duration is 5 days per PEMS set, with a total of 15 days for all three sets. 
REPORT 03.14936.12 A-25 

Bypass Setting:

a) Run NTE transient cycle using the CVS filter-based method 

b) Set bypass to produce CVS filter-based average brake-specific of about 0.025 g/hp-hr 

c) Determine the average PM mass concentration 

d) Run t he 80 S S C ummins c ycle t o c apture P M c oncentration a t e ach m ode us ing t he A VL 

MSS

e) Check the PM concentration levels and select the six-steady state modes from the 80 poi nt 

matrix. As a first order, check the concentration at the pre-selected steady-state modes to see 

if they spread within reason around the concentration produced for the NTE transient cycle. 

If not, adjust the bypass as needed to establish the right spread in brake-specific emissions 

and concentration for the six steady-state modes

f) Make s ure t hat t he poi nts s elected spread around a br ake s pecific l evel and concentration 

level of a threshold of 0.025 g/hp-hr, and concentration range of 4 to 15 milligram per cubic 

meter.

3.2.4 Data Analysis 

Use the acquired data to create the “error surfaces” to be used by the Monte Carlo simulation. 

An example of the steady-state error surface determination is shown in Table 5 for PM. 

TABLE 5. EXAMPLE OF SS ERROR SURFACE

Error Surface for SS PM Concentration

Figure 5

x-axis PM μg/mole (lab mean at setpoint)

y-axis PM μg/mole (PEMS)

Figure 6

x-axis PM μg/mole (lab mean at setpoint)

y-axis 5th percentile 5th [PM μg/mole (PEMS) - PM μg/mole (lab)]

y-axis 50th percentile 50th [PM μg/mole (PEMS) - PM μg/mole (lab)]

y-axis 95th percentile 95th [PM μg/mole (PEMS) - PM μg/mole (lab)]

The 5th, 50th and 95th percentiles from the (PEMS - lab) delta data will be used to 

estimate the 1st and 99th percentiles from assumed Gaussian distributions.

Figure 7

x-axis PM μg/mole (lab mean at setpoint)

y-axis ic_SS_PM

z-axis = 

ΔSS_PM_μg/mole

1st Percentile from Gaussian distribution based on 5th and 

50th [PM μg/mole (PEMS) - PM μg/mole (lab)] deltas. 

99th Percentile from Gaussian distribution based on 50th and 

95th [PM μg/mole (PEMS) - PM μg/mole (lab)] deltas. 

50th Percentile based on [PM μg/mole (PEMS) - PM μg/mole 

(lab)] deltas.

ic sample frequency once per NTE event trial

ic sample distribution Gaussian (normal distribution)
REPORT 03.14936.12 A-26 

3.3 PRECISION ERRORS UNDER T RANSIENT E NGINE OPERATION (DYNAMIC 

RESPONSE)

3.3.1 Objective

The objective of this portion of the work is to determine the precision error, ΔTR PM m , with each 

PM-PEMS under NTE transient engine operation. This will be achieved by creating a 20 to 25-

minute transient NTE cycle where the PEMS measure in each NTE. 

3.3.2 Background 

PEMS are expected to operate in a repeatable manner over NTE events as short as 30 s econds. 

Two sources of PEMS precision error are hypothesized: 1) dynamic response to rapidly changing 

signals, and 2) susceptibility to “history” effects. Dynamic response error includes error due to 

measurement signal time alignment, and the dissimilarity of the dynamic response and aliasing 

of s ignals; i ncluding t hose s ignals us ed t o d etermine e ntry i nto a nd exit f rom t he NTE z one. 

History effects i nclude the e ffects of p reviously measured qu antities on c urrently m easured 

quantities. F or e xample, this m ay be c aused b y i neffective s ample ex change i n the P M 

emissions sampling volumes, or it may be caused by one or more sensors’ characteristic rise time 

or fall time. To account for any dynamic response precision error, the increase in precision error 

incremental t o the s teady-state e missions me asurement pr ecision will be inc orporated into the 

overall error model. 

Selection of short NTE cycles (each 32 seconds) maximizes the sensitivity of this test to effects 

of dynamic response. Thirty-two seconds was chosen as the minimum instead of thirty seconds, 

which is the shortest NTE event time, to ensure that 1 H z ECM updating of torque and speed 

values w ould be unl ikely to i nterfere with c apturing N TE e vents. For each r epeat of t he t est 

cycle, the order of the 30 different NTE events will be the same. In addition the 29 di fferent 

intervals separating each NTE event from the next will have a r ange of durations and these will 

be randomly arranged in each test cycle as well. Fixed arrangement of the NTE events and the 

inter-NTE events w ill maximize the s ensitivity of thi s te st to dynamic r esponse a nd history 

effects, and make the DPF and bypass operation very consistent.

The total length of the NTE transient cycle will assume that only 5 quartz crystal of the Sensors 

PPMD are working, and it takes five minutes of stabilization time for reusing a crystal after PM 

collection. Thus, the same NTE transient cycle used in the gaseous PEMS program will be used 

here, except for changes in the inter-NTE times to accommodate the Sensors PPMD. 

3.3.3 Methods and Materials 

a. Use a transient engine dynamometer emissions laboratory.

b. Use a l aboratory t hat can accommodate at l east three P EMS, their pow er s upplies, the 

PEMS flow meters, cables and lines.

c. Use s ame over all guidelines de scribed i n s ection 3.2, but applied t o t ransient e ngine 

testing. 

d. Record the EEPS’ total mass signal during transient testing.
REPORT 03.14936.12 A-27 

Challenge PEMS to 30 different 32-second NTE events, shown in Table 5, over about 23 minute 

test c ycle, or w hatever needed to a ccommodate t he ne ed f or f ive c rystals of t he P PMD t o be 

operational. Randomize the NTE events shown in Table 6 once, scale up every fifth inter-NTE 

time, shown in Table 7, to accommodate the PPMD, and use the same order for repeat testing. 

Repeat the test cycle 10 times for each set of three PEMS. N ote that for any torque command 

that is less than zero, command closed throttle (i.e. zero or minimum fuel command), and motor 

the engine at the commanded speed for that data point. An example of an NTE transient cycle is 

shown in Figure 9. 

Based on 10 repeats with each set of P EMS, the t otal num ber o f r epeats w ill be 30 cycles, 

assuming 1 NTE cycle x 10 repeats x one exhaust configuration x 3 sets of PEMS x one engine 

(1x10x1x3x1 = 30). Assuming a 25 minutes of NTE with 30 minutes of forced regeneration and 

preparation for the second repeat, the total number of days for NTE transient testing is 10 days (8 

hours per day). This time includes PEMS and engine setup, PEMS warm up, and daily checks. 

Prior t o executing t he first repeat, setup each PEMS and stabilize engine ope ration at t he first 

inter-NTE ope rating poi nt. S etup t he P EMS a ccording t o 40 C FR P art 1065 a nd P EMS 

manufacturer instructions, including any warm-up time, zero-spans of the analyzers and the setup 

of all accessories including flow meters, ECM interpreters, etc. Then, when the test cycle starts, 

switch the PEMS’ to sample emissions from the engine. When the text cycle ends, switch the 

PEMS back to ambient sampling. Complete all post-test lab and PEMS validations according to 

40 CFR Part 1065 and according to PEMS manufacturer instructions. 

3.3.4 Data Analysis 

Discard from further data analysis an y NTE events invalidated by an y criteria in 40 CFR Part 

1065 Subpart J. For each NTEi event (i=1 to 30), which was repeated 30 times per engine with a 

specific e xhaust c onfiguration ( j = 1 t o 30) , calculate t he t ransient m edian a bsolute de viation, 

MADTRi mPM , for , where for each NTEi event, MADTRi = median[| NTEij – median (NTEij) |].

Next cal culate t he di fference of M AD b y s ubtracting a cor responding s teady-state M AD, 

MADSSi mPM for . MADTRi-SSi = MADTRi – MADSSi. To determine a corresponding MADSSi, 

calculate t he P EMS M ADSS at each steady-state me dian lab va lue, a nd t hen us e t he m edian 

PEMS N TEi value a long t he m edian l ab va lue’s a xis t o f ind M ADSSi for t he c orresponding 

MADTRi . D o not e xtrapolate a ny M ADSSi beyond the mini mum or ma ximum me dian lab 

values. Note that some MADSSi values might be zero because the lab data for that median failed 

the F-test in the previous section. 

For any MADTRi-SSi less than zero, set that MADTRi-SSi equal to zero. 

Create a transient error surfaces using all of the MADTRi-SSi. Be sure to include any MADTRi-SSi

data points that are equal to zero because they will affect the 1st and 99th percentile values. 

 
REPORT 03.14936.12 A-28 

TABLE 6. NTE TRANSIENT CYCLE 

NTE 

Event 1Speed % Range 2Torque % Range Description

NTE1 17% 332% Steady speed and torque; lower left of NTE

NTE2 59% 332% Steady speed and torque; lower center of NTE

NTE3 Governor line 332% Steady speed and torque; lower right of NTE

NTE4 17% 66% Steady speed and torque; middle left of NTE

NTE5 59% 66% Steady speed and torque; middle center of NTE

NTE6 Governor line 66% Steady speed and torque; middle right of NTE

NTE7 17% 100% Steady speed and torque; upper left of NTE

NTE8 59% 100% Steady speed and torque; upper center of NTE

NTE9 100% 100% Steady speed and torque; upper right of NTE

NTE10 Lower third 332% - 100% Highly transient torque; moderate transient speed

NTE11 Upper third 332% - 100% Highly transient torque; moderate transient speed

NTE12 Middle third 332% - 100% Highly transient torque; moderate transient speed

NTE13 17% - governed Lower third Highly transient speed; moderate transient torque

NTE14 17% - governed Upper third Highly transient speed; moderate transient torque

NTE15 17% - governed Middle third Highly transient speed; moderate transient torque

NTE16 Lower right diagonal Transient; speed increases as torque increases

NTE17 Upper left diagonal Transient; speed increases as torque increases

NTE18 Full diagonal; lower left to upper right Transient; speed increases as torque increases

NTE19 Lower left diagonal Transient; speed decreases as torque increases

NTE20 Upper right diagonal Transient; speed decreases as torque increases

NTE21 Full diagonal; lower right to upper left Transient; speed decreases as torque increases

NTE22 Third light—heavy-duty NTE event

from International, Inc. data set Sample from LHDE

NTE23 Cruise; ~ 50 mph Sample from HDDE

NTE24 Cruise; ~ 75 mph Sample from HDDE

NTE25 Small bulldozer Sample from NRDE

NTE26 Large bulldozer Sample from NRDE

NTE27 Second of three NTE events in FTP Seconds used from FTP: 714-725, 729-743, 751-

755

NTE28 Third light—heavy-duty NTE event 

from International, Inc. data set Sample from LHDE

NTE29 First of two NTE events in NRTC Seconds used from NRTC: 423-430, 444, 448-450, 

462-481, increased 464 speed from 40% to 42%

NTE30 First of two NTE events in NRTC Seconds used from NRTC: 627-629, 657-664, 

685-696, 714-722

1 Speed (rpm) = Curb Idle + (Speed % * (MTS - Curb Idle)

2 Torque (lbf-ft) = Torque % * Maximum Torque At Speed (i.e. lug curve torque at speed)

3 Torque (lbf-ft) = Maximum of (32 % * peak torque) and the torque at speed that produces (32 % * peak 

power)
REPORT 03.14936.12 A-29 

TABLE 7. DYNAMIC RESPONSE INTER-NTE EVENTS

INT 

Event

Duration 

(s) 1 Frequency Description

INT1 10 1 Initiation of cycle; INT1 is always first

INT2-6 2 5 Shortest and most frequent inter-NTE events

INT7-10 3 4 Short and frequent inter-NTE events

INT11-14 4 4 Short and frequent inter-NTE events

INT15-18 5 4 Short and frequent inter-NTE events

INT19-21 6 3 Short and frequent inter-NTE events

INT22 7 1 Medium inter-NTE event

INT23 8 1 Medium inter-NTE event

INT24 9 1 Medium inter-NTE event

INT25 11 1 Medium inter-NTE event

INT26 13 1 Long inter-NTE event

INT27 17 1 Long inter-NTE event

INT28 22 1 Long inter-NTE event

INT29 27 1 Long inter-NTE event

INT30 35 1 Longest inter-NTE event

INT31 5 1 Termination of cycle; INT31* is always last

Interval speeds and torques are not identical, but they are clustered around zero torque and the 

speed at which 15% of peak power and 15% of peak torque are output.

FIGURE 9. EXAMPLE OF A NTE CYCLE

Torque-Speed Domain

-100

0

100

200

300

400

500

600

700

0 500 1000 1500 2000 2500 3000 3500 4000 4500

Speed (rpm)

Torque (lbf-ft)

Lug Curve

NTE S line

NTE T-S line

1 SS Lower Left

2 SS Lower Mid

3 SS Lower Right

4 SS Mid Left

5 SS Mid Mid

6 SS Mid Right

7 SS Upper Left

8 SS Upper Mid

9 SS Upper Right

NTE Cycle

22 LD NTE3

23 OH 50mph Cruise

24 OH 70mph Cruise

25 NR Small Dozer

26 NR Large Dozer

27 FTP NTE2

28 LD NTE6

29 NRTC NTE1

30 NRTC NTE2
REPORT 03.14936.12 A-30 

3.4 ECM TORQUE AND BSFC

3.4.1 Objective

Compare t he E CM-based t orque a nd f uel r ate w ith t hat of t he l aboratory-based measurement 

using the Cummins 80 SS mode cycles. For the laboratory purposes, use the gas-based fuel flow 

values instead of the measured fuel flow. Repeat the Cummins 80 SS cycle three times, and use 

the average values produced.

Use at least six engines for these experiments that include the one engine to be used in the PM 

PEMS program and Engine B, C, and D of the ACES program. 

3.4.2 Data Analysis 

Use the acquired data pooled and normalized to % of max torque and % of maximum fuel rate to 

replace t he m anufacturer s ubmitted e rror s urfaces t hat w ere pr eviously us ed i n t he g aseous 

portion of the Monte Carlo simulation. Refer to section 2.4 f or description and example of an 

error surface. Include any bias error, unless there is an assignable cause that would not occur inuse and the steering committee approves to eliminate such bias error. 

4 ENVIRONMENTAL CHAMBER 

The envi ronmental cha mber t ests cha llenge P EMS t o a va riety o f e nvironmental di sturbances, 

namely electromagnetic interference, atmospheric pressure, ambient temperature and humidity, 

and shock and vibration. 

During e ach of t he t ests, pl us a ba seline t est, the P EMS w ill c ycle t hrough s ampling f our 

different di lution pr eparations of a erosol pa rticles t hat c ontain vol atile h ydrocarbon a nd 

elemental c arbon using a pa rticle generator tha t mimic s the f ormation of di esel pa rticles. The 

OC/EC w ill be us ed to determine t he conc entration levels ne eded for t he P M g enerator. 

Essentially, after de termining the s teady-state p oints t o r un on t he e ngine, t he O C/EC s emicontinuous ins trument w ill be us ed along w ith the f ilter-based m ethod. T hen, f or t he 

concentration levels to be used with the PM generator, the OC/EC instrument will be used to set 

the PM generator to produce the desired composition and concentration levels, similar to those 

encountered under steady-state. T hree particle concentration levels of 5, 10, a nd 15 m g/m3

, as 

shown in Table 8, will be generated by the particle generator. Each concentration will be fed to 

the PEMS after applying dilution ratios of 6, 12, 20, and 30. For each concentration and dilution 

ratio combination, the PM generator will be stabilized for 4.5 minutes, and data will be collected 

by the PEMS for 30 seconds. The test will continue for a period of 8 hours. The first six cycles of 

every test will serve to be the baseline before any environmental change is made.

The t emperature/humidity and pr essure t ests a re de signed t o mimic r eal-world e nvironmental 

disturbances w ith t he m agnitude a nd f requency of t he di sturbances a djusted t o r eal-world 

conditions. Randomly sample a uniform distribution of probability for their ic. , from any minute 

of the test. By randomly sampling from the minutes of these tests the magnitude and frequency 

of the real-world error will be built into the error model, which is described in Section 2. T he 

other e nvironmental t ests r epresent t he f ull r ange o f pos sible c onditions. F or t hese t ests, 

randomly sample the normal distribution in Figure 1 for their ic. 
REPORT 03.14936.12 A-31 

TABLE 8. CONCENTRATION AND DILUTION RATIO SCHEDULE WITH PM 

GENERATOR

Raw PM Concentration, 

μg/m3

Dilution Ratio

DR1 DR2 DR3 DR4

6 12 20 30

Concentration at Above Dilution Ratio

5000 833.3 416.7 250.0 166.7

10000 1666.7 833.3 500.0 333.3

15000 2500.0 1250.0 750.0 500.0

For E MI/RFI and vi bration, t he i nstruments w ill be s ubjected t o s creening t ests w ith H EPA 

filtered air t o detect i f t here an y cha nges i n the r esponse of t he i nstruments. Based on these 

results, the MASC will decide if the particle generator will need to be used with these tests. 

For the vibration screening test, in order to avoid damage to the instruments, a frequency sweep 

will be us ed at low a mplitude. The ide a he re i s to detect th e f requency that ma y tr igger a 

response by the instrument, without doing any damage due to high amplitude. 

4.1 DATA ANALYSIS FOR ENVIRONMENTAL TESTS

Reduce d ata b y first c alculating means for each 30 -second period of stabilized measurements. 

Subtract f rom each mean the respective ba seline con centration. T he r esults ar e errors or 

“deltas”. C orrect e ach of t hese er ror di stributions b y r emoving their r espective b aseline 

variances, which were determined b y quantifying PM Generator output with no e nvironmental 

perturbations. C alculate t he va riance of each of t he di stributions. S ubtract t he respective 

baseline variance from each calculated variance. Use the resulting difference in variance as the 

target va riance f or a djusting t he e rror di stributions. If t he t arget va riance i s z ero or ne gative, 

leave all error values of the distribution as is and do not proceed to the next step. If the target 

variance is positive, iteratively solve to find a single numerical value that can be used to divide 

each error in a given distribution such that the resulting distribution has a variance equal to the 

target variance. Now each of the errors is corrected for baseline variance.

Then, calculate the NTE result with all errors, including torque and flow errors set to zero. This 

is the true value. Then subtract the true NTE value from the result with all errors and record this 

difference i n one of t he 7 m easurement a llowance di stributions: mPM times t hree cal culation 

methods (torque-speed, fuel-specific * BSFC, ECM fuel flow) times three PEMS manufacturers, 

except Sensors and Horiba can not use the ECM fuel flow calculation method. Then proceed to 

the next NTE event in the nominal data set. R epeat the entire nominal data set over and over 

until all 7 measurement allowance distributions converge. Follow the data reduction steps set out 

in Section 2 to select the final measurement allowance.
REPORT 03.14936.12 A-32 

4.2 PM GENERATOR COMMISSIONING

The P M g enerator i s de veloped by E PA. The P M g enerator can create various h ydrocarbon 

mixtures along with solid particle generation using carbon rods arcing. The PM generator is also 

equipped with a micro-proportional diluter, and is intended to simulate diesel exhaust particle 

phase compounds. 

EPA will ship the PM generator to SwRI. EPA (Matt Spears) will train SwRI staff on using it. In 

addition, SwRI together with EPA may incorporate to it a soot particle generation mechanism 

that i s di fferent t han t he c arbon r od a rcing, using i nstead a pr opane f lame m ini-CAST 

technology. 

The PM generator will be used during atmospheric chamber testing, temperature and humidity 

testing, and may be used during EMI/RFI and vibration experiments. 

4.3 BASELINE

4.3.1 Objective

The ba seline variance will be e stablished us ing a n 8 hour ba seline t est i n w hich t he PM 

generator cycles through the same compositions and concentrations of PM used during the actual 

environmental tests. M ean values will be determined from the first five cycles through the PM 

concentrations. D eviations ( deltas) from t hese mean va lues dur ing s ubsequent c ycles t hrough 

the c oncentrations w ill be us ed t o d etermine the ba seline va riance. This va riance w ill be 

subtracted from the environmental test results. 

4.3.2 Background 

All of the ot her environmental te sts inhe rently inc orporate the ba seline bi as v ariance o f the 

PEMS. B ecause t he M onte C arlo s imulation m odel a dds all the e rrors de termined from the 

various environmental tests, it would add the baseline variance of PEMS to the model too many 

times. In order to compensate for this in the model, the baseline variance of PEMS is determined 

and subtracted from each of the environmental tests’ results. 

Note that the baseline variance of PEMS is measured and modeled (i.e. added) once as part of 

the steady-state engine dynamometer laboratory experiment.

4.3.3 Methods and Materials 

For thi s e xperiment us e a w ell ve ntilated EMI/RFI s hielded r oom c apable of m aintaining 

reasonably constant temperature and pressure. Use a room that can house one of each PEMS, 

their power supplies, the PEMS flow meters, cables and lines. 

Prior t o executing t he b aseline t est, s etup each PEMS and stabilize the P EMS in the r oom. 

Perform P EMS s etup a ccording t o 40 C FR P art 1065 S ubpart J a nd P EMS m anufacturer 

instructions, including any warm-up time, and audit. Then supply the PEMS’ sample ports with 

the sequence of PM from the PM generator as described at the beginning of Section 4. 
REPORT 03.14936.12 A-33 

At each PM concentration, flow PM long enough so that stable readings of the PEMS can be 

recorded. W hen t he O C/EC a nalyzer i s us ed t o s pot-check t he out put of t he P M g enerator, 

ensure that enough time has elapsed to achieve an accurate OC/EC analysis. 

Position PEMS and configure PM transport tubing to minimize transport delays and PM losses. 

Test at least one PEMS from each PEMS manufacturer.

4.3.4 Data Analysis 

Reduce t he b aseline d ata f or each P M P EMS, using artificial N TE s ampling e vent time s. 

Subtract from each mPM the mean mPM from the initial (short) baseline test of six cycles through 

the PM concentrations, which were conducted at the beginning of the test. The results are errors 

or “deltas”. Calculate the variance of these values, and use them for baseline variance correction 

in the data reduction of the remaining environmental tests. 

4.4 ELECTROMAGNETIC RADIATION

4.4.1 Objective

Evaluate t he ef fect of Electromagnetic Interference (EMI) and Radio frequency Interference 

(RFI) on t he performance of the PEMS and determine error factors for the PEMS due to these 

effects. First, a screening test on each instrument will be performed with HEPA filtered air to 

determine if the EMI/RFI affects the instrument response. If it doe s, the MASC will decide on 

the test matrix required for this evaluation. 

4.4.2 Methods and Materials 

Use an EMI test facility capable of running the SAE tests listed above. This would include: 

Signal generators, Power amplifiers, Transmit antennas, Electric Field Sensors, Measurement 

Receiver, Data recording device, LISNs (Line Impedance Stabilization Networks) and shielded 

enclosure.

4.5 ATMOSPHERIC PRESSURE

4.5.1 Objective

Evaluate the effects of ambient pressure on PEMS PM concentration outputs. 

4.5.2 Background 

PEMS are expected to operate over ranges of ambient pressures. It is hypothesized that some of 

the errors of the PEMS concentration outputs may be a function of ambient pressure. Therefore, 

this experiment will change the ambient pressure surrounding PEMS to evaluate its effects on 

PEMS m easured c oncentrations a nd f low m eter t ransducer out puts. A s w ith a ll of t he 

environmental tests, the test c ycle f or thi s test is ba sed on t he be st-known di stribution of r eal 

world conditions. For this test, the test cycle pressure distribution was matched to the county-bycounty annual average atmospheric pr essure di stribution i n E PA’s 2002 N ational E missions 
REPORT 03.14936.12 A-34 

Inventory (NEI) model. Figure 10 depicts the NEI data distribution (based on 3149 data points) 

and the test cycle pressure distribution. 

FIGURE 10. PRESSURE HISTOGRAM

4.5.3 Methods and Materials 

Use a barometric chamber that can be well ventilated and capable of controlling a wide range of 

pressure changes (82.74 to 101.87 kPa). Use a chamber that can house at least three PEMS at a 

time, one of each PEMS manufacturer, their power supplies, the PEMS flow meters, cables and 

lines, plus the PM generator. 

Follow a pattern of first soaking the PEMS at a constant pressure, then ramp the pressure to a 

new pressure, soak the PEMS at that new pressure, and then ramp to another pressure. Use the 

sequence of pressures and times, as shown in Table 9 along with Figure 11, to simulate a typical 

distribution of real-world pressures and changes in pressure, which are believed to be dominated 

by changes in altitude during driving in the United States.

Pressure Histograms

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

84.1 86.8 89.6 92.3 95.0 97.8 100.5

Pressure (kPa)

Frequency (% of total time)

2002 NEI Pressure Distribution <5500 ft

Test Cycle Pressure
REPORT 03.14936.12 A-35 

TABLE 9. ATMOSPHERIC PRESSURE TEST SEQUENCE

Atmospheric Pressure Test Sequence

Phase

Pressure Time Rate

kPa Alt. ft. min ft/min Comments

1 Soak 101 89 10 0 Flat near sea-level

2 Ramp 101-97 89-1203 20 56 Moderate hill climb from sea level

3 Soak 97 1203 20 0 Flat at moderate elevation

4 Ramp 97-101.87 1203- -148 60 -23 Moderate descent to below sea 

level

5 Soak 101.87 -148 20 0 Flat at extreme low elevation

6 Ramp 101.87-101 -148-89 20 12 Moderate hill climb to near sea 

level

7 Soak 101 89 20 0 Flat near sea level

8 Ramp 101-97 89-1203 20 56 Moderate hill climb from sea level

9 Soak 97 1203 25 0 Flat at moderate elevation

10 Ramp 97-96.6 1203-1316 20 6 Slow climb from moderate 

elevation

11 Soak 96.6 1316 20 0 Flat at moderate elevation

12 Ramp 96.6-82.74 1316-5501 20 209 Rapid climb to NTE limit

13 Soak 82.74 5501 20 0 Flat at NTE limit

14 Ramp 82.74-96.8 5501-1259 30 -141 Rapid descent from NTE limit

15 Soak 96.8 1259 20 0 Flat at moderate elevation

16 Ramp 96.8-90 1259-3244 15 132 Rapid hill climb to mid elevation

17 Soak 90 3244 10 0 Flat at mid elevation

18 Ramp 90-96.8 3244-1259 20 -99 Rapid descent within middle of 

NTE

19 Soak 96.8 1259 20 0 Flat at moderate elevation

20 Ramp 96.8-99.2 1259-586 20 -34 Moderate descent to lower 

elevation

21 Soak 99.2 586 20 0 Flat at lower elevation

22 Ramp 99.2-101 586-89 10 -50 Moderate decent to near sea-level

23 Soak 101 89 20 0 Flat near sea-level

FIGURE 11. PRESSURE-TIME ENVIRONMENTAL TEST CYCLE

Pressure-Time Environmental Test Cycle

-9000

-7000

-5000

-3000

-1000

1000

3000

5000

7000

9000

0 1 2 3 4 5 6 7 8

Time (hr)

Altitude (ft) & dA/dt (ft/hr)

71

75

79

83

87

91

95

99

103

107

111

115

119

123

127

131

Approximate Pressure (kPa)

______ Altitude (ft) _ _ _ _ 1/2-hr moving avg dA/dt (ft/hr) ______ Vertical gridlines = hours ______ Vertical gridlines = 7-min gas

 cylinder cycle times
REPORT 03.14936.12 A-36 

Prior t o e xecuting t his pressure s equence, s etup e ach P EMS a nd s tabilize t he P EMS i n t he 

chamber’s first pressure. P erform PEMS setup according to 40 C FR Part 1065 S ubpart J and 

PEMS manufacturer instructions, including any warm-up time, zero-span-audits of the analyzers 

and the setup of all accessories including flow meters, ECM interpreters, etc. T hen supply the 

PM P EMS’ s ample por t w ith the s equence o f P M f rom t he P M g enerator as de scribed at t he 

beginning of Section 4. 

Flow each generated PM sample long enough so that at least 30 s econds of stable readings are 

recorded for the slowest responding gas concentration output of all the PEMS. P osition PEMS 

and c onfigure gas t ransport t ubing t o m inimize t ransport d elays. T arget t o s ample about 30 

seconds. Repeat this cycle over the 8-hr test cycle, by cycling through the concentration shown 

in T able 8, w hich r epresents one hour of t esting, us ing a 4.5 m inutes of s tabilization a nd 30 

seconds of sampling at each condition. 

Perform this test once for one set of PEMS with as many PEMS tested at once. 

 

4.5.4 Data Analysis 

Perform data analysis according to Section 4.1. 

4.6 AMBIENT TEMPERATURE AND HUMIDITY

4.6.1 Objective

Evaluate the effects of ambient temperature and humidity on P EMS PM concentration outputs. 

The histogram i n F igure 12, a long w ith T able 10 a nd F igure 13, w ill be upda ted b y a ne w 

temperature profile that takes into consideration the data generated by CE-CERT.

4.6.2 Background 

PEMS ar e ex pected t o operate ove r a wide r ange of changing ambient t emperatures. It i s 

hypothesized t hat s ome of t he e rrors of t he P EMS out puts may be a f unction of c hanges i n 

ambient te mperature. T herefore, this experiment w ill c hange the a mbient te mperature 

surrounding P EMS t o e valuate i ts e ffects on P EMS m easured c oncentrations and flow m eter 

transducer outputs. As with all of the environmental tests, the test cycle for this test is based on 

the be st-known di stribution of r eal w orld c onditions. F or t his t est, t he t est c ycle t emperature 

distribution was matched to the hour-by hour county-by-county average atmospheric temperature 

distribution, weighted by vehicle mile s tr aveled a ccording to EPA’s 20 02 National E missions 

Inventory ( NEI) m odel. F igure 12 d epicts t he NEI da ta di stribution ( based on ov er 900,000 

temperatures and over 270 trillion vehicle miles) and the test cycle temperature distribution.
REPORT 03.14936.12 A-37 

FIGURE 12. TEMPERATURE HISTOGRAM

4.6.3 Methods and Materials 

Use a w ell ventilated room capable of controlling a w ide range of temperature changes (-23 to 

100°F). U se a room t hat c an hous e at l east s ix P EMS, t heir pow er s upplies, t he P EMS f low 

meters, c ables and l ines, pl us s even di fferent z ero, a udit, and s pan gas c ylinders, and a gas 

switching system.

Follow a pa ttern of f irst s oaking t he P EMS a t a c onstant r oom t emperature, t hen ramping th e 

room t emperature t o a n ew t emperature, soaking the P EMS at t hat new t emperature, and then 

ramping to another temperature. Use the following sequence of temperatures, shown in Table 10

and F igure 13 , and times t o simulate t he r ange of r eal-world temperatures and changes i n 

temperature:

Temperature Histograms

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

28.5 39.5 50.5 61.5 72.5 83.5 94.5

Temperature (F)

Frequency (% of total time)

Real World Temperature 23 -100 F

Test Cycle Temperature
REPORT 03.14936.12 A-38 

TABLE 10. AMBIENT TEMPERATURE TEST SEQUENCE

Ambient Temperature Test Sequence

Phase

Temperature Time Rate

°C °F min °C/min Comments

1 Soak 13.89 57 10 0.00 Cool in-garage pre-test PEMS operations

2 Ramp 13.89-5.00 57-23 5 -3.78 Leaving cool garage into cold ambient

3 Soak -5.00 23 5 0.00 Operating at cold temperature outside of 

vehicle

4 Ramp -5.00-12.78 23-55 145 0.12 Diurnal warming during cool day

5 Soak 12.78 55 40 0.00 Steady cool temperature during testing

6 Ramp 12.78-28.33 55-83 5 3.11 Return to hot garage on a cool day

7 Soak 28.33 83 52 0.00 Hot in-garage pre- post- test PEMS 

operations 

8 Ramp 28.33-37.78 83-100 5 1.89 Leaving ho garage into hot ambient

9 Soak 37.78 100 8 0.00 Operating at hot temperature outside of 

vehicle

10 Ramp 37.78-22.22 100-72 100 -0.16 Diurnal cooling during hot day

11 Soak 22.22 72 60 0.00 Steady moderate temperature during 

testing

12 Ramp 22.22-13.89 72-57 5 -1.67 Return to cool garage on a moderate day

13 Soak 13.89 57 40 0.00 Cool in-garage post-test PEMS operations

FIGURE 13. TIME SERIES CHART OF AMBIENT TEMPERATURE TEST

Temperature-Time Environmental Test Cycle

-60

-50

-40

-30

-20

-10

0

10

20

30

40

50

0 1 2 3 4 5 6 7 8

Time (hr)

Temperature (C)

-76

-64

-52

-40

-28

-16

-4

8

20

32

44

56

68

80

92

104

116

Temperature (F)

_____ Temperature _ _ _ _ 1/2-hr moving average dT/dt (C/hr) _____ Vertical gridlines = hours _____ Vertical gridlines = 7-min gas cylinder cycle times
REPORT 03.14936.12 A-39 

Prior to executing this temperature sequence, setup each PEMS and stabilize the PEMS in the 

chamber’s first temperature. Perform PEMS setup according to 40 CFR Part 1065 Subpart J and 

PEMS manufacturer instructions, including any warm-up time, zero-span-audits of the analyzers 

and the setup of all accessories including flow meters, ECM interpreters, etc. 

Run t he 8 -hour c ycle t est b y s tepping t hrough t he c oncentration a nd di lution r atio s hown i n 

Table 8.

 

4.6.4 Data Analysis 

Perform data analysis according to Section 4.1. 

4.7 ORIENTATION AND VIBRATION

4.7.1 Objective

Evaluate the effect of vehicle vibration on the performance of the PEMS and determine error 

factors for the PEMS due to these effects. Prior to doing extensive vibration work, perform a 

screening using HEPA filtered air sampling at a sweep of different frequencies with low 

amplitude. If any of the PEMS shows a response to a particular frequency, propose a frequency 

test and submit it for the MASC for approval. 

5 SWRI CVS AND CE-CERT TRAILER CORRELATION

Prior to performing the in-use work with the PM-PEMS, it is important to establish the degree of 

correlation between SwRI C VS-based PM m easurement and CE-CERT CV S-based PM 

measurement. For t his pur pose, t he C E-CERT tr ailer w ill move to SwRI f acilities a nd PM 

measurement will be conducted on the engine used for the PM-PEMS program. 

Prior t o t he c orrelation w ith S wRI, t he C E-CERT M EL w ill c onduct a 1065 a udit of t he P M 

measurement s ystem an d associated weighing c hambers and stations an d associated electronic 

sensors a nd m onitors. T his a udit w ill i nclude ve rification of t he s econdary di lution f low a nd 

temperature c ontrollers. T he s ampling s ystem w ill be c hecked t o m ake s ure i t hol ds t he 

appropriate temperatures and within the appropriate limits. The filter holders will be checked for 

compliance a nd t he l og books w ill be e xamined t o e nsure t hey a ppropriately m onitor a ll t he 

parameters ne eded f or 1 065 c ompliance. C E-CERT w ill i dentify ar eas where t he cu rrent C ECERT pr ocedures or equipment doe s not m eet the 106 5 r egulations a nd w ill upg rade t hese 

systems or procedures so that they are compliant with the 1065 regulations. 

The C E-CERT’s ME L w ill be c ross-correlated with an engine cel l at S wRI us ing an engine 

selected by the SC. Testing will be conducted under the NTE transient cycle with the 0.025 g/hphr bypass setting as determined by the SC.

5.1 METHOD AND MATERIALS

Below is a list of a step by step approach for the correlation between SwRI and CE-CERT
REPORT 03.14936.12 A-40 

1. Perform a propane check on SwRI CVS and 47 mm filter and CE-CERT CVS and 47 

mm filter. Both s ystems should pass Part 1065 on propane. However, even if they 

pass, note any difference between the two.

2. Set the CVS flow rate to be the same on both systems 

3. Set the filter face temperature and velocity to be the same on both systems 

4. Set the secondary dilution ratio to be the same on both systems. 

5. Use W hatman P TFE m embrane f ilters ( 7592-104), a nd f ilter s creens t hat m eet t he 

latest Part 1065. 

6. Modify the exhaust path to SwRI CVS to be comparable with that for the CE-CERT 

Trailer

7. Since SwRI is using a test cell that may have had various PM levels, both SwRI and 

CE-CERT s hould pr econdition on t he s ame e ngine. T hus, i t i s r ecommended t hat 

CE-CERT c lean t heir t unnel pr ior t o t raveling t o S wRI s o t hat bot h c an be 

conditioned on a similar emissions level.

8. Pre-condition t he S wRI C VS t unnel a nd t he C E-CERT tr ailer C VS tu nnel f or a 

period of 10 hour s a t e ngine r ated pow er us ing e xhaust c onfiguration w ith D PF 

without bypass. The conditioning time may include active DPF regenerations. 

9. Run a total of 12 repeats of the NTE transient cycle using DPF with Bypass Level at 

0.025 g/hp-hr emission level, over a period of three days. Four repeats per day with 

the CE-CERT followed by four repeats with SwRI CVS and then alternate. Prior to 

each set of four repeats manually regenerate the DPF.

10. Use S wRI DM M-230 a nd C E-CERT D MM-230 t o m ake s ure t hat t he e ngine P M 

source is not shifting and being consistent. 

11. SwRI s hould h andle a nd w eigh a ll t he f ilters for bot h S wRI and C E-CERT i n 

accordance with their protocol. 

12. The CE-CERT trailer is needed at SwRI for at least two weeks per engine. One week 

for setup and two weeks of testing assuming the above schedule. 

13. In a s eparate t ask, EPA w ill equi librate and pre-weigh 20 f ilters us ing E PA’s 

weighing protocol. EPA will then ship them to SwRI for repeat preweighing using 

their protocol. S wRI w ill then ship the s ame f ilters to EPA f or r eweighing. After 

reweighing at EPA, EPA will ship the filters to CE-CERT for weighing using CECERT’s w eighing pr otocol. F inally, CE-CERT w ill s hip the f ilters to E PA f or 

reweighing. R esults will be reported by EPA for MASC discussion. N o threshold 

for acceptance has been established at the time of this testplan writing.

14. The t arget f or c orrelation a t t he 0.025 g /hp-hr le vel is C E-CERT’s m ean of 12 

repeats being within +/-10% of the mean value reported by SwRI. 

6 MODEL VALIDATION AND MEASUREMENT ALLOWANCE DETERMINATION

The pr e-validated measurement allowance va lue f or bot h or at l east one P EMS w ill b e 

determined pr ior t o t he i n-use model va lidation at C E-CERT. If bot h P EMS s ystems ha ve 

determined reasonable measurment allowances, then the validation testing will be performed on 

the PEMS that shows the lowest measurement allowance.

The MA SC decided to validate only one of the complete PM PEMS systems Horiba or Sensors 

where AVL will “piggy back” on either PEMS as part of the model validation. Thus the testing is 

a full set of tests where three model PEMS from one manufactures over three routes with one 

bypass setting and one vehicle will be tested. If the selected model PEMS does not validate the 
REPORT 03.14936.12 A-41 

MA S C ha s t he opt ion t o t ry va lidating t he second m anufactures P EMS. T his a dditional 

validation i s not c overed i n t he C E-CERT s cope of w ork a nd w ould r equire a budg et 

modification.

6.1 MODEL VALIDATION

6.1.1 Objective

The objective of the validation testing is to validate the Monte Carlo model by 

1. Testing the PEMS in parallel with the CE-CERT trailer

2. Checking the data to see if it fits the model predicted based on the SwRI laboratory 

efforts

For the mode l va lidation testing effort, CE-CERT w ill c onduct pr eliminary pl anning for t he 

PEMS installation and commissioning. For each PEMS model tested, a total of 5 t est days are 

allocated for commissioning. Subsequent PEMS commissioning of like models should take less 

time and thus only 3 t est days are allocated. The PEMS commissioning will be performed with 

the as sistance of t he P EMS m anufacturer on s ite. CE-CERT w ill procure W hatman filters for 

both CE-CERT & Horiba filter weighing process.

CE-CERT will design, construction, and install a bypass. CE-CERT will purchase the parts for a 

bypass. The bypass will be “tuned” to the BSPM level requested by the committee. Initially, it is 

planned to tune to 0.025 g/hp-h at clean DPF condition, which could give a range of values from 

0.01 to 0.04 g /hp-h depending on i n-use conditions and DPF regeneration status. The use of a 

PM PEMS may be incorporated into this part of tuning to provide instantaneous feedback on the 

PM level in addition PM filters will be used to determine actual level. This tuning data will be 

made av ailable t o the M A S C as addi tional P EMs-MEL d eltas, but be denoted as p reliminary 

tuned data since values could exceed the desires of the MA program. 

Long line lengths will be employed to ensure good mixing. CE-CERT will use good engineering 

judgment to determine if good mixing is established. CE-CERT will evaluate good mixing b y 

measuring the real time PM with the AVL PEMS while attempting to traverse the exhaust stack. 

Given the limitation to work around a vehicle during in-use testing, traversing the exhaust may 

require s ome t ype of al ternative t est pr ocedure once t he b ypass i s f abricated. CE-CERT w ill 

work with the MA SC to determine when a suitable well mixed bypass has been achieved. 

The test matrix and test costs depend on t he actual number of PEMS tested, number of bypass 

configurations, and the number of routes. For this scope of work three model PEMS from one 

manufactures over three routes with one bypass setting and one vehicle will be tested. This test 

matrix i s ba sed on t he recommendation of t he SC. I f a s econd m anufacture P EMS r equires 

testing then a new scope of work will be needed and a budget change. 

The pr imary te sting w ill be f ocused on true N TE e vents if pr actically p ossible a nd/or f orced 

triggered events. The target level of ≥ 50 ug will be set for the filter measurements by the MEL. 

If H oriba P EMS i s c hosen t hen t he H oriba f ilter w ill be r eplaced one t ime f or e very 8 M EL 

filters to simulate an 8 hour operation for the Horiba filter. All filter weighing for both the CE-
REPORT 03.14936.12 A-42 

CERT a nd Horiba filters ( if s elected) w ill be pe rformed by C E-CERT us ing CE -CERT’s 

weighing procedures. 

For t he t est m atrix c hosen t hree m odels of one m anufacture P EMS w ill be t ested ove r t hree 

routes with one additional test day allocated, see Table below. One additional day is allocated for 

repeating a test route or for operating the PEMS in a “true” NTE mode or combination of both. 

The PEMS will be tested over each route/test-bypass configuration for a total of 4 t est days. A 

total of 10 prep days are allocated for the preparation and installation of the first PEMS for each 

manufacturer, 5 test days are allocated for commissioning each PEMS, and 3 days are allocated 

for changing between PEMS of a single manufacturer. Subsequent PEMS commissioning of like 

models should also take less time and thus only 3 test days are allocated. 

Table - Three Models of One PEMS Manufacture Test Matrix

Mfg Unit # Route Test 

Conditions

Total test 

days

PEMS 

plus AVL 

1 Palm Springs

San Diego

Baker

1 bypass 

0.025 g/hp-h 

4

PEMS 

plus AVL 

2 Palm Springs

San Diego

Baker

1 bypass 

0.025 g/hp-h 

4

PEMS 

plus AVL 

3 Palm Springs

San Diego

Baker

1 bypass 

0.025 g/hp-h 

4

Truck rental for extended period of time for setup and PEMS installation is included under this 

task. This could include a Volvo because of parts availability or a different model for ease of 

bypass installation.

Data analysis with engines outside of the NTE requires additional data processing for Method 2. 

During Method 2 calculation there is a s ummation of the inverse of fuel rate. The fuel rate on 

some conditions outside the NTE can go to zero causing the calculation to go to infinity. In these 

situations it was decided by the MASC to freeze the bsFC to a constant value during out-of-NTE 

operation using the last valid BSFC NTE value. CE-CERT will perform this bsFC freezing in the 

Method 2 summation during data post processing for both the PEMS and MEL. The logic to start 

and stop freezing will be determined by the MASC and provided to CE-CERT before processing 

Method 2 results. 

∑

∑

















×

=

Work

ECM

Carbon

g Method

fuel

fuel

2
REPORT 03.14936.12 A-43 

6.1.1.1 CE-CERT Validation

The difference between the PEMS results and the CE-CERT trailer results will be compared to 

the error predicted by the Monte Carlo model. To validate the Monte Carlo model, data must be 

run t hrough t he m odel a nd t he m odel r esults must pr edict t he a ctual t est r esults w ithin a 

reasonable level of accuracy.

Validation will be based on the following procedure. For each reference NTE event, the Monte 

Carlo model will be used to generate the 5th and 95th percentiles of the simulated distribution of 

the brake-specific PM emission differences. In order to obtain simulations representing similar 

conditions t o those obt ained on -road, some e rror s urfaces may ne ed t o be s uppressed i n t he 

simulations since not all of them ma y be applicable to the on-road conditions. T he choice of 

which error surfaces to suppress would need to be made by the Steering Committee. 

Next, the 5th and 95th delta percentiles obtained from the above simulations will be separately fit 

to a line or curve using two chosen methods: a linear regression procedure and a local regression 

(loess) technique1. Depending on which of the resulting two fits is best for each set of data (i.e., 

either for the 5th percentile deltas or the 95th percentile deltas), the resulting line or curve will be 

used as one of the lower or upper limits for the on-road data.

To determine the best fit for a given set of delta percentiles (i.e., 5th or 95th), a simple regression 

line initially will be fit to the data. If a least squares linear regression of the 5th or 95th percentile 

deltas versus the ideal PM emission has an r

2

>0.85 and an SEE < 5 % of the median ideal PM 

emissions, then the regression line will be used. If this set of criteria is not met, then a loess fit 

will be used. Since a loess regression requires the selection of a smoothing parameter2

 

to smooth 

the data, the chosen smoothness parameter should balance the residual sum of squares against the 

smoothness of the fit. 

The on -road de lta e rrors, obt ained f rom t he r esults of c ollecting da ta on s everal N TE e vents 

during on-road operations, will be plotted on a graph containing the 5th and 95th percentile delta 

limits determined from the regression fits chosen above. The graph will consist of a plot of delta 

PM versus ideal PM. The number of on-road points outside these limits will be determined and 

expressed as a percentage of the total number on on-road data points. If this number does not 

exceeds 10% of the total number of on-road data, the simulation data will be considered to be 

valid. 

6.2 MEASUREMENT ALLOWANCE DETERMINATION

6.2.1 Objective

Use t he M onte C arlo s imulation pr ogram de veloped w ith data from s ections 2, 3 a nd 5, and 

validated with section 5.1 to determine the measurement allowance for all regulated emissions, at 

2007 emissions standards. 
REPORT 03.14936.12 A-44 

6.2.2 Background 

After the Monte Carlo model has been validated and confidence in its ability to predict errors 

from PEMS instrumentation, the last step in this program will be to actually calculate a single set 

of measurement allowance for PM.

6.2.3 Methods and Materials 

Using the criteria explained in section 2.2 c alculate the various levels of measurement accuracy 

corresponding t o t he t hree P EMS m anufacturers a nd t he b rake s pecific P M e missions 

calculations. U se a ll t he va rious e rror s urfaces developed dur ing t his t est pr ogram, i ncluding 

those provided by engine manufacturers to the EPA and ARB. 

6.2.4 Data Analysis

Use the methodology explained in section 2.2, and Table 2.2 to arrive at the final measurement 

allowance.

7 TIME AND COST

7.1 TIMELINE

Table 11 is a te ntative time line pr ojecting th e ma jor ta sks to be a ccomplished during thi s 

program. The additional work if needed option is the work that may need to be done if the model 

did not validate. Otherwise, the final report will be submitted by September 30, 2009. 

TABLE 11. PROJECTED PM-PEMS TIMELINE

7.2 COST

The rough estimated cost is shown in Table 12. Based on the current estimate, a $125,000 of the 

$200,000 is needed to complete the project. 
REPORT 03.14936.12 A-45 

TABLE 12. PROJECTED COST ESTIMATE

PEMS Training, Setup, Audit, and Debug $660,000 

Steady-State and Transient Experiments $190,000.00 

SwRI and CE-CERT Correlation (1 engine) $75,000.00 

PM Generator and Environmental Testing Activities $200,000.00 

Modeling Activities (Including CO2) $225,000.00 

Data and analysis, reporting, and final report $150,000.00 

Contingency if validation fails $100,000.00 

General Contingency $200,000.00 

Grand Total $1,800,000.00 

Grand Total Without General Contingency $1,600,000.00 

Grand Total without General Contingency and 

Contingency if Validation Fails $1,500,000.00 

8 ABBREVIATIONS USED IN BRAKE SPECIFIC EQUATIONS 

Method 1: 

ePM = brake-specific emission, PM (g/hp-hr)

N = total number (of time intervals) in series

x = amount of substance fraction (mol PM/mol exhaust; note that 1μmol (emission 

constituent)/mol (exhaust) = 1ppm (part per million) 

.

n = amount of substance rate (mol/sec, in this case, mol (exhaust)/sec 

t = time interval (sec)

fn = rotational frequency (shaft), rev/min 

T = torque (N-m)

NOTE: The units of the numerator work out to gemission as is. However, using the units given 

for the denominator (RPM * N -m * s ), you would still need to divide by 1.978 to get to hp-hr 

(using RPM * N-m = kW * 9550, 1 hour = 3600 sec, and kW = hp*0.7457) 

Method 2: 

ePM = brake-specific emission, PM (g/hp-hr)

MNO2 = Molecular weight, NO2 (~46 g/mol) 

N = total number (of time intervals) in series

x = amount of substance fraction (mol PM/mol exhaust; note that 1μmol (emission 

constituent)/mol (exhaust) = 1ppm (part per million) 

~

.

n = amount of substance rate (mol/sec, in this case, mol (exhaust)/sec) that is linearly 

proportional to .

n (Note: this is a proportional sample, which means that you may use a flow 

meter that has a span error, as long as its calibration is linear)

t = time interval (sec)
REPORT 03.14936.12 A-46 

MC = Atomic weight of carbon (~12 g/mol) 

wfuel = g (carbon)/g (fuel); Note fuel is roughly 86% carbon by mass 

xCproddry = amount of carbon products on a C1 basis per dry mol of measured flow (exhaust), 

mol/mol, solved iteratively per 1065.655 

xH2O = amount of water in measured flow, mol/mol (see 1065.645 for calculations) 

efuel = brake-specific fuel consumption (g (fuel)/hp-hr)

Method 3: 

ePM = brake-specific emission, PM (g/hp-hr)

MNO2 = Molecular weight, NO2 (~46 g/mol) 

wfuel = g (carbon)/g (fuel); Note fuel is roughly 86% carbon by mass 

MC = Atomic weight of carbon (~12 g/mol) 

N = total number (of time intervals) in series

x = amount of substance fraction (mol PM/mol exhaust; note that 1μmol (emission 

constituent)/mol (exhaust) = 1ppm (part per million) 

m fuel

.

= mass rate of fuel (g/sec)

xH2O = amount of water in measured flow, mol/mol (see 1065.645 for calculations) 

xCproddry = amount of carbon products on a C1 basis per dry mol of measured flow (exhaust), 

mol/mol

t = time interval (sec)

fn = rotational frequency (shaft), rev/min 

T = torque (N-m)

t = time interval (sec)

NOTE: The units of the numerator work out to gemission as is. However, using the units given 

for the denominator (RPM * N -m * s ), you would still need to divide by 1.978 to get to hp-hr 

(using RPM * N-m = kW * 9550, 1 hour = 3600 sec, and kW = hp*0.7457) 
REPORT 03.14936.12

APPENDIX B 

STEERING COMMITTEE MEETING MINUTES
REPORT 03.14936.12 B-1 

PM Measurement Allowance Steering Committee Meeting

Meeting at CE-CERT

April 23, 24, and 25, 2008 

Future Meetings:

1. May 15-16 meeting, SwRI 

2. June 12 and 13 at for PM-PEMS and 10 and 11 for EMTC, ACS. 

3. July 29 and 30, San Antonio, SwRI 

4. August 28, 29, San Antonio, SwRI 

On-Road PM PEMS Discussion and Action Items

CO2 Activities

1. Use CO2 data pr ovided by C E-CERT dur ing t he ga seous pr ogram f or CO2 validation, 

and share the information during the May 15-16 meeting at SwRI.

Test Plan Activities

a. Explain in test plan why Horiba can’t use Method 3 

b. Replace BSFC i n t he e quation of M ethod 2 with ( fuel flow/torque *speed). 

Essentially, remove BSFC error surface

c. Show how (mp bar) for the AVL is calculated

d. For AVL Method 3, use a different error surface for PM. 

e. For reference NTEs, use the existing reference NTEs, and PM concentrations to 

produce different concentration from DPF out to threshold levels. 

f. Check section 2.3 with Bob Mason to make sure it is correct 

g. Change t he example t o PM i n the t est pl an and give an appropriate P M 

concentration range per mole basis.

h. Reexamine the text to explain the new equations better. No second by second for 

AVL

i. Change Figure 8 i n t he test pl an t o r eflect t he f uel f low i nclusion, a nd r emove 

BSFC error.

j. Change Figure 8 to update all errors that are required

k. State t he t ime and d ate b y when t he m odel could be available t o t he group, 

assuming no last minute changes are required. 

General Notes:

1. For AVL, use a 3 t o 1 dilution ratio and change the selectable range if needed but not 

the dilution ratio.

2. For the NTE windowing, use the EEPS to determine the windowing sensitivity, assuming 

a maximum of 6 seconds delay at the beginning and end. 

3. Measure the CVS dilution air temperature as close as close as possible to the exhaust and 

dilution air mixing point. This may require insulation of SwRI CVS system. 

4. Use Whatman Teflon membrane, 2 micrometer for the entire program
REPORT 03.14936.12 B-2 

5. Use the fine ambient backup screen instead of the coarse or diesel backup screen

6. List all the error surfaces in the test plan 

7. Plan Training and Comissioning of the PM Generator 

8. If 32 seconds NTE is not sufficient for the PM-PEMS, extend the length 

9. Use 10 % difference as an acceptable difference between SwRI and CE-CERT

1065 Audit (For reference see SwRI presentation)

TSI Flow Meters

• Using reference flow meters like the TSI flow meter as a transfer standards is 

okay, if independently checked via a master flow meter

Sensors PPMD

a. Verify that the external and bypass flow TSI flow meter certificates are valid for 

duration of testing 

b. Verify tha t the da ta publ ished on their indi vidual c ertificates me et the 1065 

linearity thresholds 

Horiba TRPM 

• Use total flow and dilution air flow for Horiba and make sure they meet 1065 

linearity verifications.

AVL MSS

• Meeting +/- 3 % per point on dilution ratio is acceptable. 

Vibration 

Perform a f requency s weep w ith m oderate a mplitude a nd l og t he r eal t ime s ignals f rom t he 

PEMS using a fixed PM generator level. If changes occur, design a frequency cycle around that 

frequency range to test for delta changes in PM with and without vibration. Obtain approval on 

the f requency cycle s elected from t he M ASC be fore pr oceeding. If no changes are obs erved 

during a frequency sweep, there is not a need to test for vibration. 

Use t wo o rientations ve rtical a nd ho rizontal f or t he P PMD. If t here i s enough s pace on t he 

vibration table at SwRI, test all PEMS units at once. If not, use them one by one. 

EMI

Use E MI on and of f, and s creen t he r eal t ime r esponse of t he i nstrument a t a s pecific 

concentration l evel. If a r esponse i s obs erved d uring E MI on/ off s witching, t he M ASC w ill 

decide on what the next steps are.
REPORT 03.14936.12 B-3 

Environmental Chamber:

1. Temperature and Humidity cycle will be made available by Matt Spears

2. Use a high velocity fan to blow over the 4 inch inlet section of the PPMD 

3. PM Generator Setup and Verification

4. How to do the environmental pressure test? 

a) Use three different levels concentration at the 0.02 and 0.03 g/hp-hr 

b) Run a long baseline of 8 hour s using three different toggeled PM concentration levels. 

For baseline, toggle the process five times, and use that as a baseline, followed b y the 

continuation of the baseline. 

c) Run the temperature and humidity test using the same toggeling profile. For temperature, 

humidity, and p ressure, start w ith a b aseline with f ive r epeats, and t hen ki ck of f t he 

environmental cycle.

d) A toggeling of zero, mid, and high is on the order of 15 minutes. 

There i s a pos sibility of e liminating t he z ero a nd a dd one c oncentration l evel, a nd r andomly 

sample from all the deltas generated. We need to talk to Bob Mason about this. 

Use just one of each of the PEMS for environmental activities. 

Finally, changes w ere m ade t o the t est p lan during di scussions. A lso, S wRI pr esented t hat 

attached document on the test plan. More test plan discussions will take place in the next meeting 

on May 15 and 16, at SwRI. 
REPORT 03.14936.12 B-4 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI

May 15-16, 2008 

Day 1 

1. Tim French indicates that there are several issues came up with the PM-PEMS onboard 

testing performed at CE-CERT before the CE-CERT meeting was presented.

2. Kent’s presentation, posted on FTP website: 

Some highlights, but see presentation for more details:

a. Problems with valve timing opening and closing consistency and long delays over 

3 seconds. 

b. There was one crystal that was very noisy but the software did not reject it, and 

reported data using it. 

c. A drop in frequency but an increase in mass deposit. (This is a non-issue)

d. Semtech DS 10[1].09 SP2 b5. T his beta J1939 includes filtering and was able to 

include NTEs. 

e. Post te st J1939 filtering ve rsus r eal time J 1939 filtering. Post te st c aptured 

different start up NTE than the real time filtering one. Essentially, post processing 

software needs to be scrutinized.

f. NTE yield produced by PPMD was low. Some of the valid NTE that was captured 

in PPMD post processing was rejected, after carefully reviewing the data.

g. Others. See presentation.

3. SwRI update: 

a. Horiba linearity check was resolved by using the Gilibrator directlty

b. Engine A, B, C, and D 80 steady-state testing was complete

c. Engine A was tested for 40 SS points in sub-NTE runs

d. International engine was also tested

e. All above will be posted on the website after careful review

4. Janet and Bob Presentation, please see website.

Bill Martin questioned the idea of excluding the 95th to 100th data twice. One in the error 

surfaces, and the other in the measurement allowance. He was concerned whether such 

practice will lead to truncation of the distribution. Janet will show in the next meeting 

that such practice will not lead to any truncation of data or it will effect the shape of the 

error distribution. 

5. General Discussion and Action Items: 

a. Find ways to load up the can bus during the actual testing to simulate real world 

operation by making the can bus communication busy. 
REPORT 03.14936.12 B-5 

b. Keep us ing t he gaseous error s urfaces f or M ethod 2 a nd M ethod 3 on t he P MPEMS.

c. Environmental Chamber: 

1. Use a 9 m inute c ycle for t he environmental chamber a nd go i nto t hree 

concentration levels, 3 m inutes per level. Sample for a period of 35 s econds 

from each level. 

2. Use five levels of dilution ratio ranging from 6 to 50. 

3. Check with Sensors to see if they accept an exhaust flow meter analog input. 

For M odel out put, ke ep t he 95 th as t he m easurement a llowance. H owever, i f i t di d not 

validate c onsider t he p otential of us ing ot her t han t he 95 th, if t he MASC r eaches an 

agreement on that.

Day 2: 

1. Move forward with the International Engine 

2. Communicate via J 1939 to load the engine can bus during testing. Use only the J 1939 

communication with the heavy-heavy duty diesel engine 

3. Accept the idea that Horiba will use test cell engine speed and load analog output signals

4. The right to remove outliers using good engineering judgement. 

5. Starting on June 2nd

6. Spend one week of commissioning before we start. 

with commissioning

Plot:

x-axis percent of max torque or fuel rate 

y-axis absolute difference over max torque or fuel rate

Matt’s Discussion:

Model Validation Testing at CE-CERT

a. Number of engines/vehicles

b. Number of bypass conditions, at least two, maybe 3 

c. Number of PEMS: at least two of each, highly desirable three 

d. Number of NTE events: total 100-200 per PEMS 

e. Number of route repeats: 1 to 3 

f. Types of NTE events

i. CE-CERT limits

1. Minimum filter loading: 50 micrograms
REPORT 03.14936.12 B-6 

Gaseous plus PM. But PM set priorities. Sensors goes first. 

NTE Events Route Bypass Repeat

Horiba plus AVL 30-50 Palm Springs 1 One-Run

Sensors plus AVL 30-50 Palm Springs 1, 

review

One-Run

Horiba plus AVL 30-50 San Diego 1 or 

change

One-Run

Sensors plus AVL 30-50 San Diego 1 One-Run

Horiba plus AVL 30-50 Baker 1 One-Run

Sensors plus AVL 30-50 Baker 1 One-Run

 
REPORT 03.14936.12 B-7 

PM Measurement Allowance Steering Committee Meeting

Meeting in Madison, Wisconsin 

June 12-13, 2008 

The meeting started by reviewing and approving the last meeting minutes.

PPMD Commissioning

a. Following the meeting minutes, Imad Khalek gave a presentation on the status of 

the PPMD commissioning at SwRI. The presentation was sent to the MASC, but 

was not posted on the website. 

b. David Booker also gave a presentation on the SwRI commissioning activities. The 

presentation w as s ent t o t he M ASC, but w as n ot pos ted on t he w ebsite a t t he 

request of Sensors. 

c. As a r esult of t hese t wo pr esentations, t he M ASC de cided t o g ive S ensors a 

chance t o f ix s ome of t he pr oblems e ncountered a nd c ome ba ck t o S wRI f or

additional commissioning during the week of June 16. Also, the MASC requested 

that a conference call should take place on Friday, June 20, and SwRI should give 

a status update on w hether or not the Semtech-DS/PPMD issues were fixed to a 

satisfactory level so the program can proceed. 

d. The MASC also made the following points on the PPMD: 

i. It is up to Sensors to decide on the quartz crystal equilibration time, after a 

crystal goes into a invalid NTE window. 

ii. It i s r equested that w hen all c rystals ar e l ocked out, a nd one o f t hem 

becomes available during an NTE, the crystal should wait until the current 

NTE terminates, and a new NTE starts before it samples from an NTE. 

PM-PEMS Engine Selection

Imad Khalek pointed out to the MASC the fact that the PM-PEMS program is moving 

forward with some deficiencies related to the Horiba system inability to communicate with the 

Navistar engine ECM ISO protocol. It was recommended to the MASC that a heavy-heavy duty 

diesel engine be installed in the test cell first so the Horiba s ystem can communicate with the 

engine ECM using the J1939 protocol. In addition, this will give a chance for Horiba to upgrade 

their system in preparation for the Navistar engine after a heavy heavy-duty diesel engine.

EMA ag reed to take a l ook at t he pos sibility of pr oviding a he avy h eavy-duty di esel 

engine in a timely manner. EMA agreed to make a final decision on t his issue by the June 20 

conference call.
REPORT 03.14936.12 B-8 

Horiba Concern

Horiba was concerned about the fact that Sensors brought a new model of the PPMD to be used 

on the PM-PEMS-MA program. They wanted to have a chance to update their system as well. 

The MASC decided to give Horiba a chance to work and upgrade their systems. It was decided 

that SwRI should ship back to Horiba one of their system present at SwRI, and give them until 

July 14 to ship back the system. 

Horiba was also given until September 15 t o upgrade their system so it can communicate with 

the Navistar ECM protocol using ISO-15765. 

Bill M artin made c omments a bout the di fficulty he s ees in the H oriba system me eting 1065 

requirements. Matt indicated that EPA would enter an allowance for Horiba specifically, at the 

time of Direct Final Rule (DFR), through the alternate procedure approval. 

AVL Presentation

Bill Silvis presented results on the MSS, where a compensation algorithm is added to account for 

organic carbon and sulfate. No copy of the presentation was given to SwRI for distribution. Matt 

Spears was not convinced that such compensation will be acceptable for EPA approval due to the 

significant correction required and due to the MSS principle itself. SwRI also had some technical 

reservation a bout t he pr ocess due t o i ts t echnical c omplexity. E ssentially, the pr oblem i s not 

trivial and more thorough work and understanding is still needed. 

AVL was encouraged to continue working on t his issue and refine it. It is understood that they 

will submit the compensating algorithms prior to the start of steady-state testing. They may also 

submit their compensating algorithms at any time for the MASC to have analyzed by SwRI or 

CE-CERT. 

Rey A gama s uggested t hat t ime w ould be t he be st w indow of oppor tunity f or A VL t o be 

included via DFR through alternate procedure approval. 

SwRI Test Cell

 

The MASC requested the following: 

• SwRI should install all PM-PEMS as close as practically possible to the entry 

of t he C VS. This w ill r educe an y p article l osses be tween t he poi nt of 

measurement among t he P M-PEMS and also relative t o the C VS. 

Furthermore, this will minimize the backpressure experienced by the PPMD, 

by shortening the length of exhaust piping present downstream of the PPMD, 

prior to entry to the CVS. The target backpressure is (-1 to +4 kpa) 

• SwRI should try to maintain temperature of 35°C ± 5°C in the vicinity of the 

PM-PEMS inside the test cell. 
REPORT 03.14936.12 B-9 

CE-CERT Presentation

Kent Johnson gave a presentation on the temperature distribution on various location of the MEL 

such a s be hind t he C AB, unde r M EL t rail, a nd unde r pa ssenger door . K ent’s pr esentation i s 

posted on t he w ebsite. One of i ntriguing hi ghlights of t he pr esentation is that the temperature 

profile can reach as high as 60 °C to 90 °C during in-use in the vicinity of the PM-PEMS. This 

triggers t he i dea of c hanging t he e nvironmental t emperature pr ofile t hat w as us ed dur ing t he 

gaseous PEMS program that was based on ambient temperature. Matt spears will be modifying 

the existing environmental chamber temperature profile, taking into account Kent’s finding. 

Budget

CE-CERT presented their budget with some options. The overall impression was that the total 

budget f or bot h SwRI a nd C E-CERT i s be yond t he f unding l evel a vailable f or t his pr ogram. 

Below are some of the options entertained that will be discussed during the next meeting at SwRI 

starting on July 21st

. 

1. Reduce the scope of work by cutting the number of routes and the number of PEMS used 

by CE-CERT

2. Reduce the scope of work at SwRI by reducing the number of engines to be tested from 

two engines to one engine 

3. Increase the overall budget by eliminating this year EMA pilot program requirement, and 

add funding to the MA program instead, assuming that the funding will be cost shared 

among EMA, EPA, and CARB 

Additional Comments and Action Items:

1. If PPMD and OBS200, both, resulted in negative allowance, e.g. -0.01 to -0.02 g/hp-hr, 

the MA will be zero, essentially one MA for the entire program and for all the PM-PEMS 

used. Under such circumstances, both instruments will be allowed to be used with a zero 

MA.

2. Keep aware of alignment issues. One may want to investigate the brake-specific emission 

value reported by PM-PEMS by shifting the numerator and denominators relative to one 

another and relative to absolute time. This could be done with initial transient test results 

with bypass. 

3. Sensors w ill pr ovide a VI t o simulate t he ex haust f low r ate s o we c an exercise t he 

multiple dilution ratios in the environmental chamber. 

4. SwRI will ne ed t o r esubmit s ome budge t opt ions of doi ng t he w ork w ith one e ngine 

versus two engines. 
REPORT 03.14936.12 B-10 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

July 21-23, 2008

July 21

The meeting started at 1:00 PM. It started by reviewing and approving the previous minutes, and 

also by discussing future meeting schedules, which were already sent by EMA as future meeting 

notices. T he next meeting is scheduled for August 27 ( starts at 2:00 PM), 28, a nd 29 (ends at 

2:00 PM). 

Imad Khalek started a presentation, posted on FTP website, on project status update. He showed 

a comparison between laboratory-based exhaust flow and the 4-inch Sensors exhaust flow that 

was used on the Navistar engine. The slope of the correlation was too high at 1.27. As a result of 

the discussions, the MASC decided on the following with the current 5-inch exhaust flow meters 

used w ith t he M ack engine, which t he f irst engine t o be us ed as a pa rt of t he of ficial 

measurement allowance work.

• Check laboratory-based exhaust flow rate with Horiba and Sensors 5-inch exhaust flow 

meters. If a problem is obvious or the slope of the correlation differs by more than +/- 

0.05 from a slope of 1, then send the flow meters back to the manufacturers to check on 

the calibration.

After finishing a part of the presentation, Janet Buckingham and Bob M ason showed up f or a 

scheduled presentation at 3:00 PM. The presentation is posted on the FTP website and addressed 

the double truncation issue raised by Bill Martin at the 5th and 95th percentile. The presentation is 

posted on the FTP website. Based on the presented work, the following was agreed upon: 

a. The 95th

b. The 5

percentile is still desired for the measurement allowance

th and 95th

c. There was still a remaining unresolved issue about where you assign the -1, 0,1 on 

the error surfaces for the delta change between lab and PEMS. It was decided that 

this i ssue s hould be a ddressed d uring the la st da y of the me eting, but th at w as 

never brought up a gain. There was a proposal by Bill Martin to fit both sides of 

the error distribution independently using a normal distribution fit. The median of 

such distribution will be the median based on previous practice along with 95

 are still acceptable to bound the validation range. 

th for 

one s ide and 5 th for t he ot her s ide. W hen e ach side of t he di stribution i s f itted 

using the above boundary conditions, one can then expand the data picked from 

the error surfaces to cover 1 % to 99 % of the data or even 0.1 % to 99.9 %. No 

decision has been made on this issue yet. This topic will require more discussion 

during the next meeting. 

July 22

A significant part of the day was allocated for budget discussion. The different budget scenarios 

are posted in the FTP website. Below are some of the highlights: 
REPORT 03.14936.12 B-11 

For budget cutting purposes, one engine will be used for the measurement allowance at SwRI. In 

addition, work will be performed with one bypass setting that gives brake specific PM emissions 

levels between the two thresholds of 0.02 a nd 0.03 g/hp-hr. For CE-CERT, choose the PEMS 

with the l owest pos itive m easurement al lowance. If one P EMS cl early s hows a l ower 

measurement a llowance t han t he r est, a nd bot h a re pos itive, pi ck t he one w ith t he l ow 

measurement allowance for the rest of the validation testing.

If one allowance is positive by one PEMS, and if the other one is negative, then, in principle, 

choose the positive allowance if it slightly positive. (no clear cut agreement yet).

After the budget discussion, Tim French from EMA walked the group (via phone) through the 

EMA proposal, posted on F TP website, to substitute year 1 pi lot program and to also provide 

some supplemental funding on the order of $200 K to inject more funding into the measurement 

allowance program. The overall EMA budget funding was projected to be on the order of $500 K 

to $700 K . Most or all the proposed activities will be performed in-use by hiring a third party 

that is not part of EMA and does not belong to one of the engine manufacturer to conduct the 

testing. As a result of the EMA proposal the following will take place:

• Matt Spears will speak with EPA upper-management to consider the EMA proposal in 

the context of the one year pilot program. 

• EPA will also make a final determination on whether to use the shortened version of the 

measurement allowance program. E.g. one versus two engines, by-pass, no-bypass, etc...

After the budget discussions, Imad Khalek continued his presentation that was started on the first 

day. T he w ork s howed the di fferent t orque a nd f uel errors s urfaces de termined be tween t he 

laboratory and the engine ECM public broadcast. The data covered a total of five engines (four 

heavy-heavy a nd one l ight-heavy) t hat i ncluded a D DC S eries 60, C AT C13, C ummins ISM, 

Mack MP7, and Navistar 6.4 liter engine. 

Also the work covered sub-NTE fuel flow errors. Based the sub-NTE results, it was decided that 

for f orced N TEs, i f t he engine ope ration f alls be low N TE, us e t he l ast BSFC va lue obs erved 

within the NTE. Use that only for Method 2 calculations. 

After this presentation, Imad Khalek refreshed the memory of the group by giving a status update 

presentation, posted on t he FTP website, based on the last commissioning work done at SwRI. 

The pr esentation w as p osted on t he F TP w ebsite. A fter t hat, K ent J ohnson f rom C E-CERT 

shared s ome obs ervations a bout t he S ensors da ta pr oduced b y S wRI d uring c ommissioning. 

There was no conf lict between SwRI and C E-CERT r eporting on t he r esults. K ent’s W ord 

document t hat w as s hared w ith t he group w ill be pos ted a t t he FTP w ebsite w hen i t be comes 

available to SwRI.

July 23, 2008

The focus of this day was on t he Test Plan, particularly the environmental testing. However, at 

the beginning of this day, the idea brought by SwRI earlier of adding short NTE windows for the 

laboratory t ransient t esting, along w ith s ome l ow or m edium i dle ope ration pr ior t o t he NTE 

transient cycle was discussed.
REPORT 03.14936.12 B-12 

• It was decided that Short NTE windows should be added to the thirty 32-seconds NTE 

cycle, in the inter time. A total of 10 short NTEs will be added as follows: 

1. 4 five seconds NTEs, with two back to back within an inter-time

2. 3 10 seconds NTEs 

3. 3 18 seconds NTEs 

Position two f ive s econds N TEs ba ck t o ba ck i n t he i nter-NTEs. F or H oriba, t hey 

need about 2 t o 3 seconds to exit an NTE. Thus, place the short NTEs at least five 

seconds after the end of a valid NTE to avoid continuous sampling with Horiba. 

• On the conditioning prior to the start of a transient NTE, start with five to 10 minutes at 

medium idle time before the NTE transient cycle to observe if there is an effect on the 

laboratory P M e missions r esults. If t here i s a nd e ffect, t hen pr opose an i dle t ime o r 

something similar prior to starting the NTE transient cycle. (Note that before each NTE 

transient cycle, the plan is to force-regenerate the DPF first, medium speed idle will then 

be added after the forced regeneration).

EMI/RFI/Shock &Vibration

1. For EMI/RFI, expose the PM PEMS to EMI/RFI using HEPA filtered air, similar to the 

gaseous pr ogram. T hen decide w ith t he M ASC after reviewing t he r esults, w hat i s t he 

next step. As an option, we could use the PM generator to perform this work. One PEMS 

from each manufacturer will be used for these activities.

2. Vibration and Orientation (non-road only) 

a. On t he or ientation, a sk PEMS m anufacturers on t he w orst or ientation s cenario 

postion. U se such orientation at 45o

b. Perform a frequency s weep, with a v ery m oderate am plitude, then share w ith 

MASC to decide on how to go forward. Eric should propose the sweep frequency, 

amplitude a nd dur ation. Do t wo r epeats. O ne of e ach i nstrument w ill be us ed. 

This is only for nonroad. Outside the scope of the program. 

with the appropriate orientation, and survey 

the worst case scenario for all gaseous and for PPMD, and TRPM. 

3. As for shock, ask Eric about a recommended on-highway profile for shock. 

Temperature/Humidity chamber and Pressure Chamber

For bot h t emperature a nd hum idity a nd pr essure c hamber w ork, us e t hree P M c oncentration 

levels around the two threshold levels such as 4000, 5000, a nd 6000 μ g/m3. Use a total of four 

dilution r atios of 6, 12, 20 a nd 30. A t e ach di lution r atio, s tabilize f or 4.5 m inutes a t e ach 

concentration level and measure for 30 seconds before you go to the next concentration level to 

do the same. E.g. 

1. Set the dilution ratio to 6, stabilize at 4000 μg/m3 for a 4.5 minutes 

2. Measure PM using PEMS for 30 seconds

3. Move to next concentration level of 5,000 μg/m3 and stabilize for 4.5 minutes

4. Measure PM using PEMS for 30 seconds

5. Move to the next concentration level of 6,000 μg/m3
REPORT 03.14936.12 B-13 

6. Measure PM using PEMS for 30 seconds

7. Repeat 1 through 6 at different dilution ratio

A spreadsheet on PM concentration and loading was originally made by Matt Spears during the 

meeting and was very slightly modified by Imad is posted to the FTP website

As a r esult of t he pr oposed s cenario a bove, S ensors P PMD w ill r equire a s econdary di lution 

using M PS2 t o pe rform a n 8 hour s of a ctivities us ing the a bove s cenarios. In c ase of H oriba, 

either we need to allow higher dilution ratio than those adheres to 1065 or we need to allow more 

loading on the filter past a 0.4 mg. More discussions on t his will need to take place during the 

next meeting.

The PM cycle to run by the PM generator contains five 15 minutes cycle to be used as a baseline. 

E.g. R un t he f irst f ive 15 m inutes a t nor mal t emperature, us e t hose a s a ba seline l ine, t hen 

proceed for the rest of the day to capture an eight hours of similar repeats. Do the same thing 

prior to starting the Temperature/Humidity profile and well as the Pressure Chamber profile. One 

of the remaining issues that has not been resolved around this topic is how do w e capture the 

baseline i nformation w ith t he H oriba s ystem without the ne ed to change the f ilter. More 

discussion on this subject is needed during the next meeting. 

New Temperature Profile

Take the mean of CE-CERT Cab minus ambient and add it to original temperature profile, and 

solve for new temperature and humidity profile that maintains derived from the real world data.

Perform a F ourier t ransform on t he C E-CERT Cab data, eliminate f requency content t hat ar e 

similar to the base ambient profile, and use a magic synthesizer to superimpose the frequency on 

top of the new temperature profile. Matt Spears is assigned to do this task. 

Pressure Chamber

We resolve the logistical issue of the pressure chamber work by placing the PM generator inside 

the chamber. Matt will need to send me some dimension on the PM generator to see if it fits in 

the Chamber. Other gases as well as 30 amp circuits should all be accommodated.

For Horiba, heat trace the segment of the transition to 250 o

C, similar to the PM generator outlet 

temperature for both the Horiba and AVL. 

Use each instrument separately for these tests.
REPORT 03.14936.12 B-14 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

August 27,28, & 29, 2008 

Meetings

October: 9,10-at EPA

Nov: 12,13, and 14-at SwRI

Dec.: 10,11,12-at SwRI

Discussion Points and Decisions Made

• For t he e rror di stribution c hoose t he 1 st and 99 th at -1 a nd 1, ba sed on a nor mal 

distribution fitting to extrapolate down to the 1st and 99th

• Recheck the 4 inch flowmeter calibration that was on the International engine 

. For the height of the median, if 

there is a discontinuity, choose the average or randomly pick one or the other. 

• For Horiba, filter loading is allowed to go beyond 400 m icrogram up t o 700 microgram 

or beyond as long as the flow is controlled. 

• For Horiba, do the five cycle baseline at the beginning for humidity and temperature and 

do it at the end for the pressure. This will require changing the filter. For the pressure, if 

it can be done within an hour than do it at the beginning, otherwise, do it at the end. 

• Which of the three units are acceptable to be used in the pilot program: 

 PPMD is approved 

 Horiba might be approved 

 AVL will have to make the case at EPA to see if approved

 EMA would like to know what instrument would be acceptable for pilot 

one program 

 Can EMA combine pilot efforts within companies 

 EMA will fund the additional funding required that will be required to do 

the intermediate testing that involves:

• One bypass setting, one engine at SwRI 

• Three PEMS, one manufacturer with CE-CERT

• We s hould m ove f orward on R MI, R FI, and vi bration s weep, as s oon a s w e s tart t he 

official testing 

• Due to spikes, we may revisit the post processing of the Horiba results, especially during 

the N TE tr ansient te st. As of r ight now , Horiba f irst c orrelates the e ntire E AD s ignal 

(including spike) with the filter weight (not including spikes), then apply the relationship 

for the NTE window portion of the cycle. 

SwRI Presentation and Action Items

During t he m eeting, SwRI gave a s tatus upda te vi a pr esentation, see enc losed 

presentation (Update 6), on the PPMD, MSS, and TrPM. SwRI, also discussed the bypass tuning 

for using 10 NTE transient c ycle, 80 poi nts steady-state us ing M SS, and pr ojected filter m ass 

concentration, us ing a r elationship be tween t he f ilter a nd t he M SS. In a ddition, t he b ypass 

mixing w as s hown qualitatively. Furthermore, th e tr ansient c oncentration tr ace w ith the M SS 

was presented for the 10 NTE transient cycle.
REPORT 03.14936.12 B-15 

 As a result of the work reported several action items were born: 

• Make sure that the filter-based steady-state projected concentration is consistent with the 

filter-based measured for t he s ix s teady-state poi nts s elected. T hus, pe rform an 

experiment to determine the f ilter-based m easured c oncentration f or the s ix poi nts 

selected and share the results with MASC

• Instead of s howing a qualitative r esults on t he m ixing, pr ovide s ome qua ntitative 

assessment such as a T-test.

• Based on the high spikes observed with the MSS, check if the PPMD results in a low bias 

due to the f act that it m ay be mis sing the early spike. For t hese experiments, pi ck t he 

transient N TE w indow with t he hi ghest s pike and c reate a c ycle t hat consists of 10 

repeats of that same window. During this exercise, use only three working crystals and 

vary the PPMD trigger into the NTE with time advancement of 1,2 and 3 seconds, time 

delay of 1 second, no time delay, and ECM trigger 

The above work and the problems encountered were presented to the MASC via two conference 

calls t hat w ere don e on S eptember 12 a nd S eptember 30, 2008. T he p resentations a re al so 

enclosed as Update 7, and Update 8. 
REPORT 03.14936.12 B-16 

PM Measurement Allowance Steering Committee Meeting

Meeting at EPA, Ann Arbor

October 9 and 10, 2008 

SwRI gave a status update on the progress of testing. See enclosed presentation.

There w as a con cern about the pr edicted lasting time of the P PMD in the f ield, which was 

predicted to be on t he order of 1 hour, using an average NTE threshold of 0.02 g/hp-hr. There 

was a desire to extend the running time of the PPMD to at least 2 hour s. This is indeed met by 

the ins trument if on e ta kes int o account over all collection efficiency o f the ins trument. In the 

example given, it was assumed that the collection efficiency was a 100 %, where in reality it was 

on the order of 50 %. Thus the one hour of lasting time reported is in reality two hours. 

A decision was made to move forward with testing using only the MPS1, as shown below, after 

final c ommissioning of the P PMD b y S ensors. Actually, D avid Booker f rom S ensors f lew t o 

SwRI late on October 9 to be at SwRI on October 10. 

1. Test on MPS1 only, as its currently configured at one microgram. 

2. Okay to use external trigger for steady-state testing

3. For steady-state testing:

a. Target 100 microgram on the filter

b. 50 microgram for TRPM 

c. 0.66 microgram for PPMD 

Do not clean crystals until it is apparent that the next run will likely overload the filter.

The second day of the meeting was spent at Sensors. Matt Spears gave a presentation on the PM 

generator. Later, he s howed t he P M g enerator s etup, a nd e xplained t o t he g roup t he v arious 

elements of the PM generator and the equipment used. A copy of Matt’s presentation is enclosed. 

Also, a copy of Matt’s note is shown below, particularly to 1065 PEMS changes: 

Matt’s Note:

1. Review minutes from last meeting

2. Discuss 1065 changes required for PM PEMS

3. Update from SwRI on recent activities

4. Update from PEMS manufacturers regarding recent phone conference 

5. Friday afternoon @ Sensors 

a. SUN conference presentation

b. PM Generator

6. Discuss 1065 changes required for **

a. Dilution Air

PEMS field testing only**

i. Temperature control

1. 1065 = 25±5 °C 
REPORT 03.14936.12 B-17 

2. PM P EMS= no di rect f eedback c ontrol f rom a di lution a ir 

temperature m easurement r equired-- use good e ngineering 

judgment; if directly and actively controlled then target 25 °C .

ii. You may us e a f ixed molar m ass o f th e di luted e xhaust mixt ure f or a ll 

PEMS field testing, as determined by engineering analysis.

b. “Filter”

i. Media 

1. PTFE membrane or TX-40 

ii. Face velocity

1. 1065= target near 100 cm/s actual, unless overloading 

2. PM PEMS

a. Flow-through media: (10 to 100) cm/s actual, which can be 

verified by engineering analysis

b. Non-flow through media: no specification 

iii. Temperature

1. 1065 is 47±5 °C 

2. PM PEMS target (42 to 52) at all times, with a minimum tolerance 

of 32 °C and a maximum tolerance of 62 °C, where the tolerances 

apply only during filter sampling.

iv. Conditions during mass determination 

1. 1065=see subpart B 

2. PM PEMS

a. If mass is not determined in-situ—i.e. within the PEMS—

then t he s ample c ollection m edia m ust be pr e a nd pos t 

analyzed according to 1065.190x. 

b. If mass is determined in-situ, follow .195. 

c. In s ubpart J , ha ve no r equirement t o hol d t o de wpoint 

specs for in-situ analyzers.

c. Absolute reference for inertial balance

i. Current status: QCM OEM stated specs are assumed.

ii. For 1065 m easurement allowance audit we had Sensors verify frequency 

measurement circuit.

iii. No immediate solution available

d. Cleanup 915 table for inertial batch PM analyzers: no freq, or rise/fall time specs. 

Recommend a process for determining noise, accuracy, and repeatability

e. 1065 Subpart J needs to state that field testing applies at any ambient temperature, 

pressure and humidity, unless otherwise specified in the standard setting part. 

f. State that EPA approves of electrostatic deposition technique for PM collection. 

Must meet 95% collection efficiency, as stated by the manufacturer.

g. Overall P EMS t est r equirement s hould be r eread a nd e dited t o be applicable t o 

batch analyzers. For example describe how to use a combination of steady-state 

and transient test modes to determine accuracy and repeatability separately; like

what we’re doing in the measurement allowance.

h. 1065.308-09: a lso r equired f or c ontinuous PM a nalyzers—read a nd edit 

accordingly

i. 1065 clarify that options after 400 ug loading are optional 

j. Clarify whether or not ambient air may be used for zero air for PEMS, including 

for hangup check. 
REPORT 03.14936.12 B-18 

k. Drift: allow any drift that doesn’t affect your ability to demonstrate compliance 

with the applicable standard.

7. Timetable

a. Next meetings

8. Test Plan-documentation reflecting the latest agreements

a. Validation—get from CE-CERT

b. Modeling 

9. For PEMS testing set dilution ratio based on manufacturers literature regarding maximum 

exhaust f low. Y ou m ay also us e ot her m anufacturer i nformation t o pe rform a n 

engineering analysis to estimate the maximum.

10. There will be no dilution ratio verification.
REPORT 03.14936.12 B-19 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

November 12 and 14, 2008 

Flow Alignment in PEMS

In case of continuous sampling, use a step function to measure the delay from the probe to the 

instrument. 

In case of batch sampling, use the geometry to account for time delay from the probe to the batch 

sampler

The above is done to align the flow or to account for time delay. 

Leave alone any time alignment between exhaust flow and ECM torque and speed. Use the ECM 

torque and speed to determine the integral over the NTE. 

Loss Correction

a. The principle of PM loss corrections for PM PEMS is agreed upon by the steering 

committee

b. EMA desires a legal construct in 1065 for allowing the use of PM loss corrections 

i. Open up 1065.295 to allow more types of compensating algorithms, based 

upon other variables 

ii. Utilize S ubpart J ove rall a pproval te st to validate e ntire P M P EMS, 

including its loss corrections. 

The PEMS manufacturers will be allowed to use compensation algorithms. 

• Horiba decided to use no particle loss algorithm. 

• AVL p resented a l oss a lgorithm t o c orrect f or t hermophoresis. T he l oss c orrection i s 

already defined and will be implemented via a post processor provided by AVL 

• Sensors plans to correct for particle loss and will share the process with MASC during 

the next meeting

As of today, no filter data can be shared with Sensors unless the loss correction is shared with the 

MASC.

Test Matrix, DPF out 

Full day no QCM cleaning or filter changing 

Storage, high speed, light load 

Release at peak torque, run for 20 minutes 

Store again at high speed, light load 

Release at rated power
REPORT 03.14936.12 B-20 

Do another storage at low idle, equivalent 

Cycle the PEMS every one minute to sample into NTE.

Cycle the PEMS every 32 seconds

5 second dwell time in between the NTEs 

PEMS Daily Checks Tolerance

Slope of 0.96 is acceptable by Sensors 

For Horiba, 3% on filter flow, and 3 p ercent at a dilution ratio of 5, and 5 percent at a dilution 

ratio of 15, and Mike will reconfirm. 

We will get a feedback from AVL on the accepted tolerance for a dilution ratio of 5.f 

Milestones

Milestone for the Model dry run 

PM generator milestone for commissioning, week of January 5 

Milestone for the fuel flow error surface delivery by the engine manufacturers (EMA will target 

the end of February for these data to be available. The eight or nine engines that are available 

now will be used for the dry run). 

Fixed date for the delivery of environmental chamber kit by the PEMS manufacturer. 

Ship it on the 19th of January by Sensors and Horiba. 

We will schedule EMI, RFI, on t he 26th of January, and shock and vibration on t he following 

week.

Matt Spears’ Note:

Wednesday

1. Upcoming meetings, December, January, and March all at SwRI 

a. December 10-12 at SwRI (10th

b. January 28

 2pm start, 12 2pm close) 

th – 30th (28th 2pm start, 30th

c. March 18

 2pm close) 

th – 20th (18th 2pm start, 20th

2. Review October meeting minutes

 2pm close) 

3. EPA / SwRI / CE-CERT PM filter round-robin 

a. Initial results

4. Temperature-Humidity test cycle

a. EPA cycle

b. SwRI addition of CE-CERT frequency content 

5. Status and progress at SwRI since last meeting

a. Update on mission time projections 

i. Relook at projections to see if it is possible to collect 20 failed NTE events 

( 0.02 g/bhp-hr) in one mission 

b. Test plan review

c. Decisions for MASC 

6. Budget update 

a. Arrangements for EPA/EMA/ARB
REPORT 03.14936.12 B-21 

Thursday

7. PEMS PM loss algorithms

a. AVL

b. Sensors 

c. Horiba 

8. Test plan development 

a. Inclination discussion 

9. Continue if necessary: Status and progress at SwRI since last meeting

a. Decisions for MASC 

Friday

10. Continue PM PEMS related Part 1065 changes 

11. Other meetings during our November meetings

a. Wednesday 

i. Wrap-up by 5:30pm due to another room reservation 

b. Thursday 

i. 12:00pm EMA, Rey Agama – break for lunch at 12:00pm 

ii. 1:00pm – 2:00pm, Shirish Shimpi, continue meeting

c. Friday

i. Matt Spears 10:00am to 10:45am, continue meeting 

12. Loss corrections resolutions 

a. The principle of PM loss corrections for PM PEMS is agreed upon by the steering 

committee, including EMA, EPA, ARB. 

b. EMA desires a legal construct in 1065 for allowing the use of PM loss corrections 

i. Open up 1065.295 to allow more types of compensating algorithms, based 

upon other variables. 

ii. EPA a grees tha t P M los s c orrections w ill not be a pplied to certification 

testing. If i n t he f uture E PA de sires t o a pply P M l oss c orrections t o 

certification t esting, s uch a pr ovision w ould b e pr oposed as pa rt of a 

notice of pr oposed r ulemaking be cause E PA a cknowledges t hat s uch a 

change would cause a cha nge i n the s tringency of t he ce rtification 

standard. 

iii. EPA w ill be t he a pproving bod y w ith r espect t o P M P EMS P M l oss 

correction. 

1. May make case-by-case approvals, based upon specific PM PEMS 

manufacturer circumstances, such as, but not limited to submitted, 

models, t heory, v alidation da ta, or e ven s imply t he m agnitude of 

the correction. 

2. May utilize S ubpart J overall approval t est to validate entire P M 

PEMS, including its loss corrections. 

3. May develop (with c onsultation w ith E MA) ot her pr ocedures f or 

codification within Part 1065. 

13. Sensors w ill pr ovide a n a dvance c opy of S ensors’ D ecember pr esentation, r equesting 

EMA question consolidation ahead of meeting. 
REPORT 03.14936.12 B-22 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

December 10-12, 2008 

SwRI p resented t he s teady-state t esting r esults, and the s torage and release and r egeneration 

results. The presentations were posted on the FTP site. 

Sensors pr esented t heir approach t o pa rticle l oss c orrection. A pr esentation i s a vailable at t he 

FTP site.

Craig Kazmierczak presented some in-use PM-PEMS work that was done on one of DDC trucks 

using the PM-PEMS equipment. The work was done by Sensors and presented by Craig. 

The MASC requested that the reference crystal be used during SwRI laboratory activities, if that 

is to be used on the road. 

The t esting done b y S wRI s o f ar i s acceptable. F or t he pa rticular P PMD us ed, di sable t he 

reference crystal, and use a working c rystal t o be a r eference. Essentially, operate w ith six 

working crystals, and use one for reference. 

Sensors intends to use the reference crystal for correction, but they are going to use a logic to 

decide whether or not it will be used in the post processor. 

The pos t pr ocessor s hould be a vailable be fore January 23, 2009. T he post pr ocessor s hould 

include any loss correction intended or any reference crystal correction model, remove all Part 

1065 excursions, and all other miscellaneous items that will make a measurement invalid. 

In t he current A VL pos t pr ocessor, t hermophoretic l oss w as c apped at 25 % . A VL w ants t o 

change that to include the full range of the model, and remove the 25 % cap. Both corrected and 

uncorrected data will be provided in the output of the post processor. 

We agreed to use the paired analysis for steady-state and for model validation. 

We agreed to use 5 % COV at a 1 sigma standard deviation for the CVS at 100 microgram filter 

loading. 

The a bove s ubject w as t abled f or f urther di scussion on how t o s ubtract t he C VS e rror 

contribution. Bill Martin will send his proposal to Bob and Imad. 

How to account for a regeneration in the field: 

NTE event > 30 seconds to be valid 

1-Discrete

2-Triggered by ECM 

3-The regen is defined between two regen flags 

4-If the regen occurs during an NTE event, and the length between two consecutive NTE flag x 2 

is shorter than the NTE event, then the regeneration is counted 
REPORT 03.14936.12 B-23 

5-If the regen occurs before an NTE event, and end outside the NTE event, the NTE will count. 

Tim F rench mentioned that t he s ubject of r egeneration a nd i ts i nclusion i n N TE ne ed t o be 

discussed in a different forum. 

Blow-By and how it will be computed for the measurement allowance. Next meeting

 

As of now, the results reported for storage and release and for regeneration will not impact the 

MA program. 

Matt will share the filter results during the next meeting.
REPORT 03.14936.12 B-24 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

January 28-30, 2009 

The l ast m inutes w as reviewed and a pproved. T he s cheduling for f uture m eetings and 

conference calls was as follows:

Conference Call, Thursday, Feb. 26, 2009, 9:00 am Central, length will be decided on Friday. 

Next MA-SC, April 1st(2:00 to 6:00), 2nd (9:00 to 5:00) and 3rd (8:00 to 2:00), San Antonio. 

Week of the 18th of May for EMTC and MASC (May 20, 21, 22, same as the April meeting) 

• Some of the in-use testing performed by Ce-Cert will include regeneration events. No 

action will be taken on modifying the test plan at SwRI. The test plan will remain the 

same and it will not include active regeneration.

• Bill Martin explained his proposal on the CVS variability addition to SS delta data. 

Bill Martin presented the analysis on how to assign the CVS variability into the delta 

between PM-PEMS and CVS in order to shrink the 95th and 5th A copy of his writeup is on the FTP website 

• SwRI presented work on the progress made. A copy of the presentations is posted on 

the FTP website

• Rey Agama presented an argument about using standard deviation instead of MAD 

for the data analysis. Matt Spears suggested that the MAD should be used, and if no 

validation w as obs erved a t t he e nd of t he p rogram, ot her pos sibilities c an be 

considered. The group agreed to move forward with this approach. Rey asked that we 

apply a normality test on the data, and he will consult with Bob Mason on that. This 

is in relation to applying a MAD or SD for the model. Bob Mason will present some 

material on the normality criteria during the next meeting.

• The contribution of blow-by will be a constant based on t he data presented from the 

four ACES engines at 0.00042 g/hp-hr. If the crankcase is vented to the atmosphere, 

this va lue w ill be added t o e very N TE e missions va lue. If t he c rankcase i s c losed 

loop, t he bl ow-by contribution of 0.000 42 g /hp-hr w ill not be a dded t o t he N TE 

emission values. 

• Horiba was allowed to fix a bug in their system in relation to delays between engine 

and OBS and OBS and TRPM 

• Horiba was allowed to make modifications on the ion trap voltage of the EAD for the 

third TRPM unit. 
REPORT 03.14936.12 B-25 

• The engine manufacturers agreed to submit the ECM/Lab Torque and Fuel Flow by 

March 31st, 2009. The data will not be linked to a particular manufacturer. 

• The i nstrument m anufacturers ne eds t o s ubmit t he l atest pos t pr ocessor b y F riday, 

March 6, 2009 

• The MASC agreed that the AVL measurement allowance will be performed based on 

data m easured plus t hermophoretic cor rection. The M ASC a grees t o also see d ata 

from the AVL MSS total PM prediction on a non-interference basis from the core of 

the measurement allowance program.

• The MASC was updated on the PM generator via a laboratory tour. The PM generator 

is currently set up at SwRI Particle Laboratory. 

Below is the unedited Matt Spears’ minutes:

1. January meeting agenda 

a. Next meetings?

b. EPA participation/management in MASC

c. Review of Meeting minutes

d. Regen in NTE discussion 

e. Bill Martin’s paired testing proposal 

f. SwRI data

i. Transient

ii. 2nd

iii. Lessons learned, problems? 

set of PEMS

iv. Analysis of MSS with sulfate and HC corrections 

v. ECM vs test cell torque and BSFC error surface update 

1. SwRI approach

g. PM Generator update

h. Environmental chamber update 

i. PEMS mfr readiness

ii. SwRI readiness & schedule

iii. PM Generator readiness

i. Next face-to-face meeting: April 1 afternoon, to April 3rd

j. May 13-15, placeholder-SwRI

 afternoon—SwRI

k. Bypass sizing for model validation work 

i. DDC engine

l. Filter results

m. Friday Schedule 

i. 9:00am start 

ii. Review test plan timeline

iii. Environmental chamber update 

1. PEMS mfr readiness

2. SwRI readiness & schedule

3. PM Generator readiness
REPORT 03.14936.12 B-26 

iv. AVL data with corrections 

v. Horiba general topic 

vi. Finalize agenda for February 26th meeting—finalized, finalize time. 9am 

central. 4 h rs. 12: 00pm C ST – 4:00pm C ST ( 1pm-5pm E ST) S hirish 

travelling to NRMM in Ispra Feb 26th

1. Agenda 

 

a. Make this a LiveMeeting, this is ok w SwRI 

b. Update on PEMS manufacturer post processors: 

c. Timeline upda te—SwRI: t est cel l and environmental 

chamber testing: 

1-hr

d. CE-CERT upda te on a bility t o c ome t o S wRI f or 

correlation testing & progress on bypass: 

30 min

e. SwRI data

30 min

i. PEMS s et nu mber 2: s teady-state & pe rhaps 

transient r esults—summary onl y: a nything 

remarkably different than the 1st set of PM PEMS: 

ii. Any new issues or difficulties: 

1-hr

f.

30 min

vii. PM Generator / nanoparticle lab tour

viii. Engine manufacturers to submit ECM/test cell torque/fuel rate data 

by March 31st

ix. Rudy’s for lunch

meeting
REPORT 03.14936.12 B-27 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

April 1-3, 2009 

PM-PEMS Meeting: April 1-3, 2009. 

Wednesday, 2:00-6:00 PM

• Meeting Minutes from previous meeting were reviewed

• Practice measurement allowance on incomplete set of data was presented by J anet, see 

presentation on website 

• The viability of using Method 3 w as discussed. There was a general agreement that this 

method ne eds t o be dr opped out m ainly b ecause t he l ack o f i nformation on t ime 

alignment between ECM fuel flow and gas-based fuel flow, but such decision needs to be 

reconfirmed during the next meeting.

Thursday, 9:00-5:00 PM 

• The nor mality t est r equirement w as di scussed b y B ob M ason, s ee p resentation on 

website. The d ecision f or now i s not t o a ssume normal di stribution a nd us e t he M AD 

instead of SD. If the MAD failed to produce a validated measurement allowance, the SD 

will be revisited.

• David Booker from Sensors presented the features of the new PPMD post processor, see 

presentation on website 

• Mike A kard f rom H oriba ga ve a n upd ate on t he s tatus of H oriba’s pos t processor, s ee 

presentation on website 

• SwRI gave a presentation on the following (see presentation on FTP website): 

• Status update and project progress 

• SS data from all the PM-PEMS

• Transient data with and without engine drift from all the PM-PEMS

• OC/EC data for SS

• EEPS data for SS

• Some MSS corrected data for sulfate and HC

• Reference N TE using m ethod 3 g ave di fferent va lues t han M ethod 1 and 2, a nd 

decision will need to be made on Method 3 during the next meeting. 

Friday, 8:00-2:00 

• Vibration for offroad was discussed. SwRI plans to do a vibration sweep similar to onhighway but while the PEMS sitting at the vibration table at 45 degree angle. SwRI will 

present a cost estimate for this additional activities when the vibration activities start.

• SwRI s howed t he f uel a nd t orque da ta s ubmitted b y all e ngine m anufacturers t hat 

included C aterpillar, C ummins, D etroit D iesel, Navistar, a nd V olvo P owertrain. S wRI 

will present all data together during the next meeting. 
REPORT 03.14936.12 B-28 

Some of the decisions/action items made during this meeting were as follows:

• A decision on whether or not Method 3 should be used (AVL only) needs to be made

• Reference NTEs will be tweaked so the main distribution of events will be between 

0.015 g/hp-hr and 0.035 g/hp-hr 

• SS error surfaces will be presented during the next meeting

• Transient data was drift corrected using the CVS, but the MASC desired to look at the 

integrated AVL data as compared with the CVS to look at the possibility of using real 

time AVL data for drift correction. The integrated AVL for the NTE cycles will be 

compared w ith t he i ntegrated C VS dur ing t he ne xt m eeting, be fore m aking a f inal 

okay on the CVS drift correction method 

• EPA was to provide some information on t emperatures experienced during off-road 

in-use a ctivities s o i t c an be i ncorporated w ith t emperature a nd hum idity pr ofile 

during environmental testing. EPA was to propose a final temperature and humidity 

profile for the program, after incorporating the CE-CERT and off-road data 

• Off-road vibration tests will be added at 45 degree angle 

• Fuel a nd t orque e rrors us ing t he e ngine m anufacturers’ s ubmitted f uel a nd t orque 

needs to be presented during the next meeting 
REPORT 03.14936.12 B-29 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, San Antonio

May 20-22, 2009 

PM-PEMS Meeting, May 20-22, Meeting Agenda 

1. Review of last meeting minutes

2. Next Meeting Schedule

3. Overall Project Status Update

4. CE-CERT and SwRI Correlation 

5. Update on EMI and RFI Testing 

6. Update on Environmental Chamber Testing 

7. SS Error Surfaces and Adjusted Reference NTE events

8. Other agenda items and questions 

• The last meeting minutes were reviewed

• The next meeting schedule was set to July 15-17 in Indianapolis 

• The overall project status we presented. See presentation by SwRI

• CE-CERT and SwRI Correlation was presented. See presentation by SwRI 

• EMI and RFI testing results were presented. See presentation by SwRI. No error 

surfaces will be generated. The problems mainly affected instrument functions. 

i. Horiba will investigate the issue related to Bulk Current Injection effect on 

the Horiba exhaust flow 

ii. Sensors will demonstrate a fix to the exhaust flow problem during the next 

meeting. Sensors will conduct their own testing if necessary. 

• SS error s urfaces were pr esented. Sensors data were not fully analyzed because 

the new post processor was not provided to SwRI. Sensors promised to provide a 

new pos t pr ocessor r esolving t he i ssues i dentified b y S wRI dur ing t he l ast 

meeting to increase data yield. Sensors later provided the new post processor, and 

the SS error surfaces were presented during a co nference call that took place on 

June 29, 2009. See SwRI presentation for the June 29 conference call. 

• The transient errors were also discussed, and the approach for the transient error 

surface w ill be p resented dur ing t he ne xt m eeting i n Indianapolis, a long w ith a 

final recommendation

• Sensitivity on f uel flow a nd CO2 flow f or M ethod 2 a nd M ethod 3 will be 

discussed during the next meeting in Indianapolis 
REPORT 03.14936.12 B-30 

PM Measurement Allowance Steering Committee Meeting

Meeting at SwRI, Indianapolis

July 14-17, 2009 

1. Last Minute Review

2. SwRI Presentation

a. Project Update 

b. SS and Transient Error Surfaces treatment and results

c. Environmental T esting ( Atmospheric P ressure, T emperature and H umidity) 

Results and Discussion 

i. Three approaches were presented on how to treat the error surfaces for the 

environmental testing. 

ii. Approach 3 was used. The steering committee agreed with using approach 

3 

iii. No error surfaces were obtained for Horiba and Sensors for environmental 

testing

iv. Error s urfaces f or A VL, but a nything b elow t he l owest M AD, s et t o a 

constant equal to the lowest MAD, and anything above the highest MAD 

set to a constant equals to the highest MAD 

d. For Method 3, and for the reference NTEs, we agreed that the gas based fuel flow 

will be advanced in order to match the ECM fuel flow. 

3. Rules on Measurement Allowance:

a. Pick t he pos itive m easurement a llowance t hat i s c loset t o zero ba sed on t he 

Horiba’s and Sensors’ PEMS. 

b. If bot h H oriba’s and S ensors’ P EMS ha ve a ne gative m easurement a llowance, 

pick the one that is closest to zero. 

4. Preliminary conference call scheduled for August 20, 2009. We will confirm it tomorrow.

At the August’s conference call a de cision on the measurement allowance will be made 

and an instrument will be selected to go to CE-CERT. After that, SwRI will ship the units 

to CE-CERT to arrive at CE-CERT earlier than September 1

5. Meeting at CE-CERT on September 22

st

nd to observe and check PEMS installation by CECERT.
REPORT 03.14936.12 B-31 

PM Measurement Allowance Steering Committee LiveMeeting

August 20, September 11, 2009 

1. SwRI presented the Monte Carlo Simulation (See Presented at FTP website)

2. The m easurement a llowance w as de termined b ased on M ethod 2 us ing S ensors’ 

PPMD

3. The measurement allowance was 0.00605 g/hp-hr

4. Sensors’ PPMD was chosen for CE-CERT in-use testing

5. It was agreed by the MASC that the AVL MSS will also participate in in-use testing 

along with the PPMD, just like it was done in the laboratory 

6. The MASC requested the following from SwRI 

a. plot the Sensors and Horiba 95th

b. show the results of the simulation based on reference data available within the 

concentration range obtained in the laboratory 

 delta on the same plot 

c. plot the 95th, 50th, and 5th

d. Refreshment on the regression rules for the validation deltas 

, for the validation 

7. Address Item 6 above in a Livemeeting on September 11 

8. The requests in Item 6 above were addressed in a livemeeting on S eptember 11 ( see 

presentation on FTP website) 

a. The M ASC r equested t hat t he va lidation de ltas f or t he 95 th, 50th, a nd 5 th

b. The LOESS fit should be done on Sensors’ validation deltas based on Method 

1 and Method 2. 

be 

regressed using the LOESS f itting r ule s ince the c riteria s et f or l inear 

regression is not met.

c. SwRI should present the regression the CE-CERT September 22 meeting.
REPORT 03.14936.12 B-32 

PM Measurement Allowance Steering Committee Meeting

February 17-18, 2010, US EPA, Ann Arbor 

Sensors gave a presentation, proposing a correction factor for the exhaust flow. 

• Sensors calibrated the flow meter in house with the same wrong pressure configuration in 

the field 

• Sensors de termined a c orrection f actor of 1.52 for e xhaust f low c orrection for U nit 3. 

This correction factor was accepted by the MASC

• A s imilar c alibration will be done on Unit 2 to de termine the e xhaust f low c orrection 

factor to be used for this unit 

Sensors i ndicated t hat a dditional c orrections r elated t o b ypass f low t hat w as not c orrected f or 

during the in-use testing is needed:

• The sample flow during NTE is determined as the difference between bypass flow before 

sampling and bypass flow during sampling. 

• If t he b ypass f low i s n ot c orrected f or ba rometric pr essure dur ing s ampling, a nd t he 

barometric changes during the long NTEs, then there would be an error introduced to the 

sample flow. 

AVL di scussed t he a pproach t hey t ook t o c alculate t he br ake-specific P M e missions us ing 

Method 2. The ECM broadcast fuel term in the equation was essentially frozen if it went below 

5% of the max fuel flow encountered during the test. However, to move forward with Method 2 

calculation, the MASC agreed that to the following:

• The MAX fuel flow provided by the engine manufacturer should be used not the max 

fuel determined during a test. Thus, the engine manufacturer of the CE-CERT vehicle 

should provide the information to CE-CERT to do proper Method 2 calculation 

• The fuel flow along with the gas concentration terms in the equation will be frozen if the 

ECM fuel flow dipped below 10% of the MAX fuel flow. 

CE-CERT gave a pr esentation on the different exhaust flow correction attempts they made. See 

CE-CERT presentation at the FTP website.

SwRI pr esented va lidation f or P PMD U nit 2 and 3 us ing M ethod 1 with a n e xhaust f low 

correction of 1.52. T he P PMD ba sed on M ethod 1 f ailed t he va lidation c riteria. S wRI a lso 

showed different scenarios for the AVL, including Method 1, 2, and 3. The presentation is listed 

on the FTP website. 

Below is a summary of the action items for CE-CERT and SwRI as a results of this meeting. 

These action items were compiled by Chris Laroo:

Follow-up Work to Finish PM MA Test Program

1) Agreed to exhaust flow correction factor of 1.52 for unit #3. 
REPORT 03.14936.12 B-33 

2) Will use exhaust flow correction factor for unit #2 that comes out of Sensors check of the 

unit #2 flow meter on their flow stand (value anticipated to be at or near 1.52 but needs to 

be determined). Value is anticipated to be known the week of March 1st. 

3) Correct PPMD data for incorrect setting in QCM bypass flow. S oftware setting change 

from 0 to 1. 

4) Use Semtech DS barometric pressure readings to account for altitude change effects on 

data. If it is determined that the PPMD barometric pressure reading is f aulty, we will 

request t hat S ensors up grade t he s ensor t o t he q uality us ed i n t he D S. This c ould be 

deemed a special source of error and accounted for via a future hardware improvement.

5) PPMD Unit #1 data will be reported for single crystal use only to reflect that fact that the 

Mass S ensitivity w as i ncorrectly e ntered into the s oftware dur ing te sting. T his s ingle 

crystal use data will then be pooled with the multi-crystal use results from units #2 and #3 

to determine the final validation % results.

6) Report PPMD Unit #2 a nd #3 de ltas for single crystal use only as a pr obing exercise to 

see if multiple crystal use has an effect on mass loss. The intent of this exercise is to 

gauge the effectiveness of proposed fixes by Sensors to eliminate the PPMD low bias for 

future pilot and compliance program testing. T his data will be pooled with the unit #1 

single crystal use results for plotting in the validation window for experimental purposes 

only. These results will not be used to determine validation. 

7) Method #2, i f the fuel flow rate d rops below 10 % of manufacturer de clared maximum 

fuel rate value, then the ratio of the emission concentration terms and the ECM broadcast 

fuel rate will be frozen at that value.

8) CE-CERT will reprocess all PPMD and MSS data with correct factors.

9) CE-CERT w ill w rite a final r eport t o be reviewed by t he s teering committee be fore 

finalization.

10) SwRI w ill pl ace r evised CE-CERT da ta i n v alidation w indows a nd c alculate ne w 

validation percentages. CE-CERT should have the data to SwRI by the end of March. 

11) SwRI will also place revised CE-CERT single crystal usage data in validation windows 

and calculate n ew validation percentages ( for experimental purposes only). T his a gain 

will help gauge whether or not there is an effect on mass loss from multiple crystal usage.

12) SwRI w ill w rite a f inal report t o be r eviewed by t he s teering committee be fore 

finalization.
REPORT 03.14936.12

APPENDIX C 

CRYSTAL BALL OUTPUT FILE DESCRIPTIONS
REPORT 03.14936.12 C-1 

EXTRACT DATA FILES

1.0 Simulation Variables

The simulation variables listed in Table B-1 were extracted at the completion of the Monte 

Carlo simulation run for each reference N TE ev ent. C rystal Ball cl assifies va riables i nto two 

categories: assumptions and forecasts. Assumptions are the estimated inputs into the simulation 

model such as the variability indices used to sample each error surface. Assumption variables in 

this study are identified by an “ic” at the beginning of the variable name, or by “Delta” at the 

beginning of the variable name. The “ic” variables are the simulation error model inputs such as 

“01_ic_SS_PM”. The “Delta” variables serve as switches that turn a given error surface on or 

off in the model, e.g. “Delta PM SS”. The Delta switch variables when turned on and off during 

a s imulation a re applied i n pos t-simulation analysis to determine s ensitivity of results to the 

particular error surfaces. Forecasts are values calculated by a forecast formula in the spreadsheet 

cells. Examples of forecast variables used in this study are “001AVL_DePM (g/hp-hr), Method 

1” and “005AVL_Valid DePM (g/hp-hr), Method 2”.

TABLE B-1. SIMULATION VARIABLES

Variable Name Description

001AVL_DePM (g/hp-hr), Method 1 MC Delta PM Method 1 for AVL PEMS

001Horiba_DePM (g/hp-hr), Method 1 MC Delta PM Method 1 for Horiba PEMS

001Sensors_DePM (g/hp-hr), Method 1 MC Delta PM Method 1 for Sensors PEMS

002AVL_DePM (g/hp-hr), Method 2 MC Delta PM Method 2 for AVL PEMS

002Horiba_DePM (g/hp-hr), Method 2 MC Delta PM Method 2 for Horiba PEMS

002Sensors_DePM (g/hp-hr), Method 2 MC Delta PM Method 2 for Sensors PEMS

003AVL_DePM (g/hp-hr), Method 3 MC Delta PM Method 3 for AVL PEMS

004AVL_Valid DePM (g/hp-hr), Method 1 Validation M C D elta P M M ethod 1 f or 

AVL PEMS

004Horiba_Valid DePM (g/hp-hr), Method 1 Validation M C D elta P M M ethod 1 f or 

Horiba PEMS

004Sensors_Valid DePM (g/hp-hr), Method 1 Validation M C D elta P M M ethod 1 f or 

Sensors PEMS

005AVL_Valid DePM (g/hp-hr), Method 2 Validation M C D elta P M M ethod 2 f or 

AVL PEMS

005Horiba_118_Valid DePM (g/hp-hr), Method 2 Validation M C D elta P M M ethod 2 f or 

Horiba PEMS

005Sensors_Valid DePM (g/hp-hr), Method 2 Validation M C D elta P M M ethod 2 f or 

Sensors PEMS

006AVL_Valid DePM (g/hp-hr), Method 3 Validation MC D elta P M Method 3 f or 

AVL PEMS

01_ic_SS_PM

Random Sampling Variability Index for SS 

PM Error Surface, applied to AVL, Horiba 

and Sensors PEMS

02_ic_TR_PM

Random S ampling V ariability I ndex f or 

Transient P M E rror S urface, applied to 

AVL, Horiba and Sensors PEMS

04_ic_Atm.Pres_PM_AVL

Random S ampling V ariability I ndex f or 

PM A tmospheric P ressure E rror S urface, 

applied to AVL PEMS
REPORT 03.14936.12 C-2 

05_ic_Amb.Temp_PM_AVL

Random S ampling V ariability I ndex f or 

PM Temperature Error Surface, applied to 

AVL PEMS

07_ic_SS_CO Random Sampling Variability Index for SS 

CO

10_ic_Pressure_CO Random S ampling V ariability I ndex f or 

CO Pressure

11_ic_Temperature_CO Random S ampling V ariability I ndex f or 

CO Temperature

13_ic_SS_NMHC Random Sampling Variability Index for SS 

NMHC

14_ic_TR_NMHC Random S ampling V ariability I ndex f or 

Transient NMHC

16_ic_Pressure_NMHC Random S ampling V ariability I ndex f or 

NMHC Pressure

17_ic_Temperature_NMHC Random S ampling V ariability I ndex f or 

NMHC Temperature

19_ic_NMHC_Ambient Random S ampling V ariability Index f or 

Ambient NMHC

20_ic_SS_flow Random Sampling Variability Index for SS 

Exhaust Flow

21_ic_TR_Flowrate Random S ampling V ariability I ndex f or 

Transient Exhaust Flow

22_ic_Pulsation_flow Random S ampling V ariability I ndex f or 

Exhaust Flow Pulsation

23_ic_Swirl_flow Random S ampling V ariability I ndex f or 

Exhaust Flow Swirl

25_ic_Radiation_Exhaust Flow Random S ampling V ariability I ndex f or 

Exhaust Flow EMI/RFI Radiation

27_ic_Temperature_Exhaust Flow Random S ampling V ariability I ndex f or 

Exhaust Flow Temperature

28_ic_Pressure_Exhaust Flow Random S ampling V ariability I ndex f or 

Exhaust Flow Pressure

29_ic_TR_Torque Random S ampling V ariability I ndex f or 

Dynamic Torque

30_ic_Torque_DOE Random S ampling V ariability I ndex f or 

Torque Design of Experiments Testing

31_ic_Torque_Warm Random S ampling V ariability I ndex f or 

Torque Warm-up

32_ic_Torque_IP

Random S ampling V ariability I ndex f or 

Torque I ndependent P arameters H umidity 

and Fuel

34_ic_Torque_Interpolation Random S ampling V ariability I ndex f or 

Torque Interpolation

35_ic_Torque_Engine Manufacturers Random S ampling V ariability I ndex f or 

Torque Engine Manufacturers

42_ic_Fuel_Engine Manufacturers Random S ampling V ariability I ndex f or 

Fuel Engine Manufacturers

43_ic_TR_Speed Random S ampling V ariability Index f or 

Dynamic Speed

44_ic_TR_Fuel Rate Random S ampling V ariability I ndex f or 
REPORT 03.14936.12 C-3 

Dynamic Fuel Rate

45_ic_SS_CO2 Random Sampling Variability Index for SS 

CO2

46_ic_TR_CO2 Random S ampling V ariability I ndex f or 

Transient CO2

49_ic_Temperature_CO2 Random Sampling V ariability I ndex f or 

CO2 Temperature

Delta PM SS Model s witch c ontrolling P M S S e rror 

surface application

Delta PM Transient Model s witch c ontrolling P M Transient 

error surface application

Delta PM Atmospheric Pressure Model switch controlling PM Atmospheric 

Pressure error surface application

Delta PM Ambient Temperature Model s witch c ontrolling P M A mbient 

Temperature error surface application

Delta CO SS Model s witch c ontrolling C O S S e rror 

surface application

Delta CO Atmospheric Pressure Model switch controlling CO Atmospheric 

Pressure error surface application

Delta CO Ambient Temperature Model s witch c ontrolling C O A mbient 

Temperature error surface application

Delta NMHC SS Model switch controlling NMHC SS error 

surface application

Delta NMHC Transient Model switch controlling NMHC Transient 

error surface application

Delta NMHC Atmospheric Pressure

Model s witch c ontrolling N MHC 

Atmospheric P ressure er ror su rface 

application

Delta NMHC Ambient Temperature Model switch controlling NMHC Ambient 

Temperature error surface application

Delta Ambient NMHC Model switch controlling Ambient NMHC 

error surface application

Delta Exhaust Flow SS Model switch controlling Exhaust Flow SS 

error surface application

Delta Exhaust Flow Transient Model switch c ontrolling E xhaust F low 

Transient error surface application

Delta Exhaust Flow Pulsation Model s witch c ontrolling E xhaust F low 

Pulsation error surface application

Delta Exhaust Flow Swirl Model s witch c ontrolling E xhaust F low 

Swirl error surface application

Delta Exhaust EMI/RFI Model switch controlling Exhaust EMI/RFI 

error surface application

Delta Exhaust Temperature Model s witch c ontrolling E xhaust 

Temperature error surface application

Delta Exhaust Pressure Model switch controlling Exhaust Pressure 

error surface application

Delta Dynamic Torque Model switch controlling Dynamic Torque 

error surface application

Delta Torque DOE Testing Model s witch c ontrolling T orque D OE 

Testing error surface application

Delta Torque Warm-up Model switch controlling Torque Warm-up 
REPORT 03.14936.12 C-4 

error surface application

Delta Torque Humidity Model switch controlling Torque Humidity 

error surface application

Delta Torque Interpolation Model s witch c ontrolling T orque 

Interpolation error surface application

Delta Torque Engine Manuf Model s witch c ontrolling Torque ( Engine 

Manufacturer) error surface application

Delta Fuel Engine Manuf Model s witch c ontrolling F uel ( Engine 

Manufacturer) error surface application

Delta Dynamic Speed Model s witch c ontrolling D ynamic Speed 

error surface application

Delta Dynamic Fuel Rate Model s witch c ontrolling Dynamic F uel 

Rate error surface application

Delta CO2 SS Model s witch controlling C O2 S S error 

surface application

Delta CO2 Transient Model s witch c ontrolling C O2 Transient 

error surface application

Delta CO2 Ambient Temperature Model s witch controlling C O2 A mbient 

Temperature error surface application

2.0 Statistics

Descriptive statistics summarizing the values obtained during a single reference NTE event 

simulation are provided in Table B-2. 

TABLE B-2. DESCRIPTIVE STATISTICS FOR SIMULATION VARIABLES

Statistic Definition

Trials Number of times the simulation was repeated

Mean Arithmetic average

Median The value midway b etween the s mallest value a nd the largest 

value 

Mode Value that occurs most often

Standard Deviation Measurement of variability of a distribution. The square root of 

the variance

Variance The a verage of t he s quares of t he de viations of a num ber of 

values from their mean

Skewness A measure of the degree of deviation of a distribution from the 

norm of a symmetric distribution

Kurtosis A measure of the degree of peakedness of a distribution

Coefficient of Variability Standard deviation/Mean

Minimum Smallest value

Maximum Largest value

Range Width Largest value – smallest value

Mean Standard Error Standard deviation of the distribution of possible sample means
REPORT 03.14936.12 C-5 

3.0 Percentiles

Percentiles a re t he probability of achieving va lues be low a pa rticular percentage i n the 

following increments: 0, 5, 10, 15, 20, 25, 30, 35 , 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 

and 100%. Percentiles are computed for each of the simulation variables described in Table. 

4.0 Sensitivity Data

Sensitivity da ta a re pr ovided b y c omputing t he r ank c orrelation c oefficient f or a ll e rror 

surfaces and all simulation variables. T he EXTRACT data file contains the absolute value of 

the rank correlation. In post-simulation processing, values of control variable Delta PM SS in 

the simulation results were applied to dichotomize the data.

5.0 Trial Values

The value for all simulation variables is provided at each trial of the simulation.

REPORT FILES

1.0 Report Summary

This s ection includes th e s imulation start d ate and time, stop date a nd time , number o f 

trials run, sampling type (Monte Carlo), random seed used, and run statistics. 

2.0 Forecasts

Descriptive s tatistics, percentiles, and a frequency histogram a re p rovided for forecast 

variables 001AVL_DePM (g/hp-hr), Method 1 through 006AVL_Valid DePM (g/hp-hr), Method 

3 (see Table).

3.0 Assumptions

Descriptive s tatistics, percentiles, distribution parameters, and a di stribution chart a re 

provided for assumption variables 01_ic_SS_PM through 49_ic_Temperature_CO2 (see Table).

4.0 Sensitivity Charts

Sensitivity charts are provided for forecast variables 001AVL_DePM (g/hp-hr), Method 1

through 006AVL_Valid DePM ( g/hp-hr), M ethod 3 (see Table). C rystal Ball cal culates 

sensitivity by computing rank correlation coefficients between every assumption (error surface) 

and forecast (delta BSPM emissions) while the simulation is running. Positive rank correlations 

indicate that an increase in the assumption is associated with an increase in the forecast. The 

larger the absolute value of the rank correlation the stronger the relationship. 

The sensitivity charts developed during the MC simulation are displayed as ‘Contribution 

to Variance” ch arts w hich are cal culated by s quaring the r ank correlation coefficients f or all 

assumptions used in a particular forecast and then normalizing them to 100%. Figure displays a 

sensitivity c hart f or t he A VL delta P M M ethod #1. T he a ssumptions w ith t he hi ghest 

contribution t o va riance ( in a bsolute va lue) are plotted a t t he t op of t he c hart. T his e xample 
REPORT 03.14936.12 C-6 

shows a typical dominant effect of the PM SS error surface represented by the 79.5% negative 

effect of Delta PM SS. As seen in the example in Figure B-1, as you increase the SS PM there is 

an increase i n t he de lta P M M ethod #1 values r epresented by t he 7.6 % pos itive ef fect o f 

ic_SS_PM, and as you increase the torque warm-up there is a decrease in the delta PM Method 

#1 va lues r epresented b y the 2.8 % ne gative e ffect of i c_Torque_Warm. O nly t he t op e ight 

assumptions are plotted in this sensitivity chart.

FIGURE B-1. SENSITIVITY CHART FOR AVL DELTA PM METHOD 1
REPORT 03.14936.12

APPENDIX D 

MONTE CARLO SPREADSHEET COMPUTATIONS
REPORT 03.14936.12 D-1 

1.0 DESCRIPTION OF ASSUMPTIONS

 The following assumptions were made in running the Monte Carlo model: 

• Only on e r eference N TE e vent can b e r un at a t ime t hrough t he M onte Carlo 

simulation workbook. However, NTE event cases can be stacked and run in a batch 

mode. 

• Uniform (1 second in duration) time steps are used in the reference NTE events. 

• Standard f ormat and e ngineering uni ts f or r eference N TE da ta established f or t he 

project are observed, and applied to the reference NTE event before the NTE event is 

entered in the Error Model workbook for Monte Carlo simulation. 

• Any we t – dry m atter c onversions, i f not ne gligible, h ave b een p erformed on t he 

appropriate reference NTE event values before the reference NTE event was entered 

in the Error Model workbook for Monte Carlo simulation. No wet – dry conversions 

are performed in the workbook. 

• Any reference NTE event normalizations to produce similar emissions brake-specific 

results f rom the thr ee e missions calculation m ethods ha ve be en a ppropriately 

performed before the reference NTE event was entered in the Error Model workbook 

for M onte C arlo s imulation. N o no rmalizations a mong t he t hree methods a re 

performed in the workbook. 

• PM e missions models f or t hree c alculation m ethods a re c omputed f or t he A VL 

PEMS. O nly calculation methods 1 a nd 2 are computed for the Horiba and Sensors 

PEMS.

• Error surface models and supporting data were approved by the Steering Committee. 

• The error model spreadsheet has been correctly implemented, and its interaction with 

Monte Carlo tools like Crystal Ball is correctly understood.

• Random number generation by a Monte Carlo tool like Crystal Ball is correct. 

• Convergence of the completed MC simulation was processed and checked outside of 

the simulation workbook. B enchmark checks on the convergence calculations were 

made using a SAS computer program. 

2.0 WORKSHEET DESCRIPTIONS

2.1 Macro Description

The M acro c an be vi ewed i n t he E xcel s preadsheet ‘ Batch C ontrol’ with t he m enu 

selections Tools>Macros>Macro1>Edit. T he purpose of Macro1 is to control NTE event batch 

processing of stacked cases. For each NTE event case processed, the macro expedites clearing

extra cells below the reference NTE event in the spreadsheet ‘Error Model’ Methods worksheet
REPORT 03.14936.12 D-2 

and deletes extra rows i n the D elta error w orksheets. T he m acro a lso pe rforms M ode 0 

calculations and s tores r esultant ' ideal e missions' va lues for a pplication i n s ubsequent M onte 

Carlo simulation. 

The user must begin with the starter version of the ‘Error Model’ Excel file which has 

300 rows of equations in columns X - CF and in rows 52 - 351 in the Methods worksheet. The 

starter spreadsheet also has 300 rows of equations below charts in columns B – F, or B – L, in 

applicable Delta worksheets. The us er (when n ot unde r automatic b atch c ontrol) c opies the

reference NTE event into columns A – V, row 52 and down, in the Methods worksheet. It is can 

then be confirmed that cell J45 in the Methods worksheet displays the correct number of rows of 

the reference NTE event. 

Macro e xecution c an be a ccomplished t hrough t he m enu s elections 

Tools>Macros>Macro1>Run. Note that t his macro clears cel ls w ithout de leting rows i n t he 

Methods worksheet, and deletes rows in the Delta worksheets. This macro will not work if the 

reference NTE event has only one row. For a reference NTE event with exactly two rows, this 

macro will c orrupt the second " check" values i n columns B-F t ype Delta w orksheets. C heck 

values are not used in the simulation, but are provided as a diagnostic aid. Apply the macro for 

reference NTE events with no more than 300 rows. 

The reader can follow the description of execution that follows by viewing the macro and 

observing the comment rows provided throughout the macro text. In execution, the macro first 

reads the contents of J45 in the Methods worksheet. It uses the number of rows in the reference 

NTE event defined by J45 to determine how many rows to clear and delete in the spreadsheet. It 

checks that the number of rows is between 2 and 299, inclusive. It will also execute correctly for 

300 rows. 

 

Next, the macro clears cell contents in columns X – CF below the reference NTE event in 

the M ethods w orksheet. N ote t he m acro, as w ritten, will not ex ecute properly i f t he s tarter 

spreadsheet has been revised with row insertion or deletion in certain areas of the spreadsheet. 

As written, the macro initiates in cell X52, counts down through the NTE Event rows, and clears 

contents in the range from there in column X through cell CF351. 

 

Next, the macro deletes extra rows below the reference NTE event, where applicable, for 

example in Delta worksheet 07. For Delta worksheet 07 it initiates in cell B79 and counts down 

through the rows of the reference NTE event to the first row to be deleted. It selects the range of 

rows from there down through row 378, deletes the rows, copies some equations and a value to 

the last row in the range the charts use, and returns the cursor to c ell F68 leaving the display 

more or less centered on the charts in the worksheet. 

Subsequently, the macro performs similar operations in other Delta worksheets; however, 

the initiating cell and final row differ among the worksheets. The Delta worksheets processed in 

this way are 7, 10, 11, 16, 17, 20, 21, 22, 23, 29, 30, 43, 44, 45, 46 and 49. 

Following t he r ow d eletion ope rations i n t he D elta w orksheets, or di rectly w hen t he 

reference N TE e vent ha s 300 r ows, t he m acro pr epares f or t he M ode 0 ( ideal e missions) 

calculation. F irst, i n t he M ethods w orksheet i t c opies t he e quations i n r ow 52, c olumns X 

through CF, to the last row in the reference NTE event. This clears any errors introduced in the 
REPORT 03.14936.12 D-3 

last row; however, it assumes that row 52 is correct. The last cell in column AC (∆t) is cleared 

for aesthetics, since the ∆t values are not applied in the model calculations. 

The Mode 0 c alculation is performed by the macro by changing the value in cell A6 of 

the Summary worksheet t o 0. T hen i n t he M ethods w orksheet, t he va lues f rom c ells C U22 

through CU30 are pasted (values only) to cells O22 through O30 where they are referenced by 

formulas during Monte Carlo simulation. T he macro changes the value of A6 in the Summary 

worksheet to 2 in preparation for the Monte Carlo simulation, and moves the cursor to cell CT18 

of the Methods worksheet. 

Additional c omments r egarding t he m acro ope ration a re presented in the f ollowing 

section descriptions of the model spreadsheet.

2.2 Worksheet 1: ErrorControl

The E rrorControl w orksheet of t he E rror M odel w orkbook i mplements 31 logic s witch 

functions. T he user enters a numerical “1” in column AD in each row corresponding to error 

surfaces t o be i ncluded in the cal culation. A n umerical “0 ” i s appl ied to error s urfaces t o be 

excluded i n t he c alculation. Corresponding r andom va riables f or e rror-surface on -off s witch 

random effects sensitivity modeling are implemented in column W under Crystal Ball control.

Error s urfaces are num bered 1 t hrough 49. T he num bered e rror s urfaces a re de fined i n 

columns A – C, and information pertinent to their usage is presented in columns E – V of the 

worksheet. Column E displays warning messages when an unusual value is monitored in column 

D.

The c ontrol s witch e lements i n t he w orksheet a re de liberately pl aced on r ows i n t he 

worksheet c orresponding t o t he e rror s urfaces t o e xpedite e quation c hecking i n t he M ethods 

worksheet where the control switch variables are applied in conjunction with error surfaces from 

the correspondingly numbered “Delta” worksheets 

The numbered error surfaces and time alignment controls that have been implemented are 

defined in the following Table C-1. 
REPORT 03.14936.12 D-4 

TABLE C-1. ERROR SURFACES USED IN SIMULATION

Component No. Error Surface

Delta PM 1 Delta PM SS

2 Delta PM Transient

4 Delta PM Atmospheric Pressure

5 Delta PM Ambient Temperature

Delta CO 7 Delta CO SS

10 Delta CO Atmospheric Pressure

11 Delta CO Ambient Temperature

Delta NMHC 

NMHC = 0.98*THC 

13 Delta NMHC SS

14 Delta NMHC Transient

16 Delta NMHC Atmospheric Pressure

17 Delta NMHC Ambient Temperature

19 Delta Ambient NMHC

Delta Exhaust Flow 20 Delta Exhaust Flow SS

21 Delta Exhaust Flow Transient

22 Delta Exhaust Flow Pulsation

23 Delta Exhaust Flow Swirl

25 Delta Exhaust EMI/RFI

27 Delta Exhaust Temperature

28 Delta Exhaust Pressure

Delta Torque 29 Delta Dynamic Torque

30 Delta Torque DOE Testing (Interacting Parameters Test)

31 Delta Torque Warm-up(Interacting Parameters Test)

32 Delta Torque Humidity / Fuel(Independent Parameters Test)

34 Delta Torque Interpolation

35 Delta Torque Engine Manufacturers

Delta Fuel 42 Delta Fuel Engine Manufacturers

Delta Speed 43 Delta Dynamic Speed

Delta Fuel Rate 44 Delta Dynamic Fuel Rate

Delta CO2 45 Delta CO2 SS

46 Delta CO2 Transient

49 Delta CO2 Ambient Temperature

The thirty-one (31) error surfaces that have been implemented are included or excluded 

by the controls numbered 1 – 49 identified in Table. When all 31 error controls are on (included 

in calculation), the sum of column D in the worksheet ErrorControl is 31. 

2.3 Worksheet 2: Summary

The Summary worksheet in the Error Model workbook comprises input mode control in 

rows 4 – 10 and output summary in rows 88 and 119. Other rows in this worksheet are available 

for diagnostic purposes. 
REPORT 03.14936.12 D-5 

The c alculation m ode c ontrol i s a ccomplished with c ell A 6 w here t he us er nor mally 

confirms that a numerical value of “2” is designated. M ode 2 de signates emissions calculation 

with all errors applied. Mode 1 corresponds to a calculation of emissions with all errors applied 

except environmental errors. Mode 0 designates an “ideal” emissions calculation with no errors

applied. In Monte Carlo error model simulation performed in this study Mode 2 was used. 

Mode 0 i s used prior to Monte Carlo simulation to generate the “ideal” emissions for a 

given reference NTE event. The Mode 0 values are calculated by entering a value of “0” in cell 

A6. T he M ode 0 c alculation a nd subsequent storing of the “i deal” em issions results ma y b e 

accomplished m anually (as de scribed a bove) or by ex ercising a pr ovided macro. T he m acro 

automatically sets the value in cell A6 to zero, calculates and saves the “ideal” emissions values, 

and returns the value in A6 to “2” in preparation for the Monte Carlo simulation. The locations 

where the reference NTE event must be entered manually, and the locations where the “ideal” 

emissions must be saved (done automatically if the macro is used) are described in the Methods 

worksheet section. 

Mode 1 in cell A6 is not typically used but can be applied for diagnostic purposes. 

The out put s ummary s ection of t he S ummary worksheet i n r ows 88 a nd 119 presents 

numerically and descriptively labeled outputs of the emissions and emissions error calculations.

In the output summary, the cells that are highlighted in turquoise color are designated by 

Crystal Ball as “Forecast” (or output) random variables. 

A total of 14 outputs (“Forecasts”) are designated in the Summary worksheet rows 88 and 

119 covering t he num ber of out put va lues f rom PM e mission, t hree c alculation m ethods 

(Methods 1, 2 and 3) for the AVL PEMS and two methods (Methods 1 and 2) for the Horiba and 

Sensors P EMS, for t he ful l e rror m odel a nd f or t he va lidation m odel (designated Valid in 

Summary worksheet variable labels). All of these “Forecasts” are provided in units of grams/hphr. This variety of calculations was accomplished with the Methods worksheet. 

2.4 Worksheet 3: Methods

 The Methods worksheet of the Error Model workbook comprises the following areas: 

• Notes and diagnostic guides are located principally in rows 1 – 21 in columns A – CF, 

continuing on row 5 through column DD. 

• Reference NTE event d ata a re l ocated i n rows 35 - 351 of columns A – W. A ctual 

reference NTE event d ata m ust be ent ered manually (or aut omatically under ba tch 

control) starting on r ow 52 i n columns A – V. Cell W52 data must be entered, and is 

provided f or s pecial c ase s tudy w here t he M ethod 3 f low-weighted PM conc entration 

may di ffer f rom M ethods 1 a nd 2. One t o 300 rows of reference NTE event da ta are 

allowed. Uniform ( one s econd i nterval) time steps ar e as sumed r epresented by t he 

reference NTE data.

• Parameters calculated are located in rows 35 – 351 of columns X – CF. The number of 

rows of these parameter equations must match the number of rows in the reference NTE 
REPORT 03.14936.12 D-6 

event. Excess cells in these columns may be cleared manually or automatically during 

execution of the macro.

• Mode 0, Ideal Emissions for this reference NTE event are stored in column O rows 22 – 

30 (either manually or automatically by the macro). Related data on the same rows are 

located in columns CT – DD.

• Input i c random variable distributions (Crystal Ball uses the terminology “Assumptions” 

for these inputs) are located in rows 26 – 32 of columns AG - CC. 

• Emissions calculations by three methods are located in rows 6 - 81 of columns CH – CJ 

(Method 1) , C L – CN (M ethod 2) a nd C P (Method 3) . T his part of t he w orksheet 

calculates full model and validation model. 

2.5 Methods Worksheet: Notes and Diagnostic Guide 

In rows 1 – 22 for columns A – CF, several descriptive labels and references are defined 

for us e i n n avigating t hrough t he w orksheet. R ow 5 , columns A -DD, contains column 

identification num bers r eferenced i n r ows 7 t hrough 22 ( depending on t he c olumn). F or 

example, in column H the values 65 on row 8 indicates that the values in column H (rows 52 and 

following rows) are a pplied i n c olumn 65 ( BM) labeled on r ow 5. If t he us er s crolls t o c ell 

BM52 it is observed that the spreadsheet formula in the cell refers to values from column H. The 

information in the notes and diagnostic guide was not applied by the spreadsheet in any of the 

emissions c alculations. It w as i ncluded w ith the intent to simplify di agnostics b y pr oviding 

information on l ocations w here spreadsheet va lues w ere applied elsewhere i n the s preadsheet. 

Outside the a reas indicated above, some other n otes, comments and di agnostic guides may be 

found in other areas of the spreadsheet. 

2.5.1 Methods Worksheet: Reference NTE Event

The reference NTE event used in the simulation was entered in rows 35-351 of columns 

A – W. A ctual reference N TE e vent data m ust be e ntered m anually s tarting on r ow 52 i n 

columns A – V. Cell W52 data must be entered, and is provided for special case study where the 

Method 3 f low-weighted PM concentration may differ from Methods 1 and 2. A minimum of 

one and a maximum of 300 rows of reference NTE event data are allowed. Equal time steps (1 

second intervals) are as sumed in the reference NTE d ata rows. T he s tandard f ormat a nd 

engineering uni ts of reference NTE event da ta e stablished f or t his pr oject m ust be obs erved. 

These are described in the column headings on rows 47 – 51, columns A – V. 

2.5.2 Methods Worksheet: Parameters

Parameters applied in the t hree emissions m ethods ar e c alculated in rows 35 – 351 of 

columns X – CF. The number of rows of these parameter equations must match the number of 

rows i n t he reference NTE event. E xcess cel ls i n these col umns m ay b e cl eared manually o r 

automatically during execution of the macro. 

The formulas applied in rows 52 a nd down in columns X – CF have been produced by 

normal edit-copy (typically of row 52 in these columns) and edit-paste to rows 53 and following 
REPORT 03.14936.12 D-7 

rows in these columns. The ∆t values displayed in column AC are not used in any calculation, 

but are displayed so a user can confirm uniform reference NTE event time sampling. T he last 

cell in column AC can be cleared (done automatically by the macro). Note that excess cells in 

these c olumns m ust be cleared, a nd r ow de letion ope rations s hould not be a pplied s ince t his 

would affect other areas in the Methods worksheet. 

Certain s ums a re pe rformed i n s everal c olumns ove r t he Parameter r ows (range of t he 

reference NTE event). These are accomplished in row 46 in columns AI, AW, AX, BO, BQ, BU

and CA. Flow-weighted PM concentration SS and TR errors are consolidated, with and without 

environmental errors, i n t he area of c ells A N40:AB47. Certain constants appl ied in the 

calculation are s tored i n c ells A W42, BC42, BI42, BP40 and BP42. O ther c onstants or 

conversion f actors a re i ncorporated num erically i n s preadsheet f ormulas. T ypical of the se is 

“0.01” to convert a percentage to a fraction. 

Specific parameters or variables are calculated in the various columns for application in 

all three methods, full model and validation model. Table C-2 lists the parameters used in the 

Methods w orksheet, t he c olumns w here t hey are c omputed a nd a br ief de scription of t he 

parameters.

TABLEC-2. METHODS WORKSHEET PARAMETER COLUMN DESCRIPTIONS

Methods Worksheet Parameters Column Descriptions

Subject Column Description

Engine operating

state percentages

X – AB Convert NTE Event variables to percentages: speed, torque, fuel rate, exhaust flow

∆Time AC Displays ∆t between NTE Event rows

NMHC AD Calculate NMHC ppm as 0.98 of THC ppm

Fuel Rate AE Calculate fuel rate g/s based on fuel density of 851 g/L

Exhaust Flow Calculations AF Convert exhaust flow SCFM to mol/s

AG Sum exhaust flow errors from Delta tabs 20, 21, 22, 23, 25 27 and 28 expressed in % 

of mol/s maximum. Respective ErrorControl tab switches are applied.

AH Convert the total exhaust flow error in % of maximum mol/s to mol/s

AI Add the mol/s exhaust flow error to the exhaust flow in mol/s. Mode control logic is 

applied.

Speed with error AJ Add engine speed error from Delta tab 43 expressed as % of engine range converted 

to engine speed in rpm. Mode control logic and ErrorControl switch are applied. 

Fuel rate with error AK Combine Delta tab 42 Fuel (engine manufacturer) with fuel rate from D elta tab 44 

expressed a s % o f maximum fuel r ate converted t o g /s t o en gine fuel r ate i n g/s. 

Mode control logic and ErrorControl switch are applied.

Torque AL Sum torque errors from Delta tabs 29, 30, 31, 32, 34 expressed as % of peak torque, 

and from Delta tab 35 e xpressed as % of NTE point torque converted to % of peak 

torque. ErrorControl switches are applied.

AM Add the total torque error expressed as % of peak torque converted to N·m to engine 

torque in N·m. Mode control logic is applied.

PM, μg/mol AN42, 

AN43

Sum P M SS a nd TR e rrors from D elta t abs 1 a nd 2 f or AVL P EMS e xpressed a s 

μg/mol, AN 42 for Methods 1 and 2, AN43 for Method 3. ErrorControl switches are 

applied.

AO42 Sum PM SS and TR errors from Delta tabs 1 and 2 for Horiba PEMS expressed as 

μg/mol, for Methods 1 and 2. ErrorControl switches are applied.

AP42 Sum PM SS and TR errors from Delta tabs 1 and 2 for Sensors PEMS expressed as 

μg/mol, for Methods 1 and 2. ErrorControl switches are applied.

AQ42, 

AQ43

Sum AVL f low-weighted P M concentration to a ll e rrors e xcept environmental P M 

errors, A Q 42 f or M ethods 1 a nd 2, A Q43 f or M ethod 3. M ode c ontrol l ogic i s 

applied. 

AR42 Sum Horiba flow-weighted PM concentration to all errors except environmental PM 
REPORT 03.14936.12 D-8 

errors. Mode control logic is applied.

AS42 Sum Sensors flow-weighted PM concentration to all errors except environmental PM 

errors. Mode control logic is applied.

AQ46, 

AQ47

Sum AVL flow-weighted PM concentration to all errors including environmental PM 

errors Delta tabs 4 a nd 5, AQ 46 for Methods 1 and 2, AQ47 for Method 3. M ode 

control logic is applied.

AR46 Same formula as AR42 as no environmental PM error model applied to Horiba flowweighted PM concentration.

AS46 Same formula as AS42 as no environmental PM error model applied to Sensors flowweighted PM concentration.

Speed · Torque AW Form product of Speed (rpm, all errors case, column AJ) and Torque (N·m, all errors 

case, column AM) for application in Methods 1 a nd 3. C onvert rpm to radians/sec 

with 2 πradians/revolution, m inutes t o s econds w ith 60s ec/min, N ·m/sec to w att h r 

with 3600Joules/watt hr, and watt to kw with 1000w/kw.

AX Form product of Speed (rpm, no errors for validation case, column O) and Torque 

(N·m, no e rrors for validation case, column T) for application in Methods 1 a nd 3. 

Convert r pm to r adians/sec w ith 2 πradians/revolution, m inutes t o s econds w ith 

60sec/min, N ·m/sec to watt h r w ith 3 600Joules/watt h r, a nd w att to k w with 

1000w/kw.

CO and ∆CO, % AY Sum environmental CO errors including errors from Delta tabs 10 and 11.

AZ Sum other CO errors. Error from Delta tab 7 is the only one developed.

BA Add the total CO errors expressed as % to engine CO in %. Mode control logic is 

applied.

NMHC and ∆NMHC, ppm BE Sum environmental NMHC errors including errors from Delta tabs 16, 17 and 19.

BF Sum other NMHC errors including errors from Delta tabs 13 and 14.

BG Add t he t otal N MHC e rrors e xpressed a s ppm t o e ngine N MHC i n PPM. M ode 

control logic is applied.

CO2 and ∆CO2, % BK Sum environmental CO2 errors. Error from Delta tab 49 is the only one developed.

BL Sum other CO2 errors including errors from Delta tabs 45 and 46.

BM Add the total CO2 errors expressed as % to engine CO2 in %. Mode control logic is 

applied.

Exhaust Flow ·

[ NMHC + ( CO +CO2 

BO

) ] / 

[fuel mass flow rate / Speed 

· Torque ]

Form product of hydrocarbons fraction plus CO and CO2 fractions (all errors case, 

columns BG, BA and BM) and exhaust flow (mol/s, column AI) divided by ratio of 

fuel r ate ( g/s, al l er rors c ase, co lumn AK) t o s peed-torque p roduct ( all er rors cas e, 

column AW) for application in PM Method 2. 

BQ Form product of NMHC fraction plus CO and CO2 fractions (all errors case, columns 

BG, BA and BM) and exhaust flow (mol/s, column AI) divided by ratio of fuel rate 

(g/s, no errors case, column AE) to speed-torque product (no errors case, column AX) 

for application in PM Method 2 validation. 

NMHC + ( CO+CO2 ) BS Form sum o f NMHC fraction plus CO and CO2 fractions ( all er rors cas e, co lumns 

BG, BA and BM) for application in Method 3. 

Fuel R ate / [ N MHC + ( 

CO+CO2 

BU

) ]

Form quotient, Fuel Rate (g/s, all errors case, column AK) divided by sum of NMHC 

fraction plus C O a nd C O2 fractions ( all er rors cas e, co lumn B S) f or ap plication i n 

Method 3. 

Fuel R ate / [ N MHC + ( 

CO+CO2 

CA

) ]

Form quotient, Fuel Rate (g/s, no errors case, column AE) divided by sum of NMHC 

fraction plus C O a nd C O2 fractions ( all er rors c ase, c olumn B S) for a pplication i n 

Method 3 validation. 

2.5.3 Methods Worksheet: Mode 0 Ideal Emissions 

For the reference NTE event in rows 52 and down in columns A – V, an ideal emissions

value must be calculated and stored for application in the emissions difference calculations. The 

ideal cas e c an be cal culated either m anually, or a utomatically b y the ma cro. F ollowing the 

calculation, the i deal v alues ar e s tored by edit-copy e dit-paste-special-values ope ration t o t he

cells i n c olumn O , r ows 2 2 – 30. T he m anual ope rations de scribed below ar e p erformed 

automatically by the macro, if executed, after manually entering the reference NTE event. 
REPORT 03.14936.12 D-9 

After manually entering the reference NTE event to be simulated and checking that the 

number of rows of equations in the Parameters section matches the rows in the reference NTE 

event, a num erical “0” can be ent ered i n c ell A 6 of t he S ummary w orksheet. T he M ethods 

worksheet should h ave calculated M ode 0 r esults us ing t he reference NTE event. If e rror 

messages like “#VALUE or #DIV/0!” are displayed, there is probably still a m ismatch between 

the rows of the reference NTE event and Parameter equations. When calculated properly (with 0 

in Summary A6), the values displayed in the Methods worksheet columns CU, CV and DB will 

be e qual on e ach of t he r ows 22 – 30. T he va lues i n c olumn CX a re not yet e qual ( unless 

previously calculated and stored for this reference NTE event) be cause they reflect t he values 

stored i n M ethods worksheet column O, r ows 2 2 – 30. T he ne xt m anual s tep i s t o edit-copy 

column CU, rows 22 - 30, and store the values by edit-paste-special-values in column O, rows 22 

- 30. Now in rows 22 - 30 the columns CU, CV, CX and DB should be equal. The final step is 

to return to Summary w orksheet cell A6 and change the value from 0 t o 2. A t this point the 

spreadsheet c ould be r un i n M onte C arlo s imulation t o pr oduce pr operly s ampled va lues. 

However, i f t he us er de sires t o m onitor charts pr ovided i n t he D elta worksheets dur ing t he 

simulation, f urther r ow-matching to the reference NTE event i s r equired in most of the D elta 

worksheets. 

The manual operations described in the previous paragraph are intended to explain how 

the Mode 0 ideal emissions are calculated and stored for use in the Monte Carlo simulation when 

∆emissions values a re calculated using th e ide al emissions r esults s tored i n O 22 – O30. T he 

reference NTE event m ust be e ntered w ith an o peration s uch as a m anual edi t-copy and e ditpaste or edi t-paste-special-values op eration. T he m acro a utomatically performs t he m ode 0 

calculation, stores the mode 0 results in O22 – O30, and changes Summary A6 back to mode 2. 

The macro also deletes extraneous rows from all the appropriate Delta worksheets so the 

charts t herein di splay pr operly. It i s i mportant to c opy t he reference NTE event i nto a f ully 

‘loaded’ starter file with equations filled on 300 rows in the Parameters area, and with full 300 

row complement of equation-rows in each of the appropriate Delta worksheets for the macro to 

modify the spreadsheet properly. 

 

2.5.4 Methods Worksheet: Input ic Random Variable Distributions

Probability distribution parameters are applied, and simulation trial values of the inputs 

are generated i n rows 2 6 – 32 o f c olumns A G - CC. R ows 26 a nd 27 a re us ed t o i nput 

distribution parameters. Rows 28 and 29 contain descriptive labels brought from the appropriate 

Delta w orksheet. Row 30 i s a n i nformation-only num ber, row 31 c ontains t he na me l abel 

applied i n M onte C arlo s imulation to the inpu t i c, and r ow 32 is w here t he M onte C arlo 

simulation tool places generated randomly-sampled values during simulation. The values in row 

32 are r eferenced by f ormula i n the r espective D elta w orksheets w here t hey are us ed for 

interpolation on the error surfaces. 

The M onte C arlo t ool in Crystal B all us es the te rminology “Assumptions” f or the se 

inputs. Two distribution forms are applied: truncated normal (Gaussian), and discrete uniform. 

For t he nor mal di stribution, t he applied s tandard de viation i s i n row 27 . In C rystal Ball, t he 

standard deviation cell on row 27 and the label cell on row 31 were referenced by equation in the 

Crystal Ball a ssumption s etup w indow, t he m ean w as i nput as 0, a nd t he di stribution w as 

truncated at -1.414319083 and at +1.414319083. Since all the truncated normal i c distributions 
REPORT 03.14936.12 D-10 

are identical (although the sampled trial v alues from each will be random in the Monte Carlo 

simulation), the Crystal Ball define-copy data and define-paste data operations were applied to 

define t he t runcated no rmal di stributions f or ot her i c variables on ce t he f irst one ha d be en 

defined. 

For the discrete uniform distributions, the minimum discrete value (1 in all cas es) was 

applied i n r ow 26, t he m aximum di screte va lue w as appl ied in r ow 2 7 a nd t he ot her r ow 

descriptions ar e t he s ame as before. A gain, on e of t hese i nputs w as s etup w ith C rystal B all 

“define assumption” and then applied with Crystal Ball define-copy data and define-paste data 

operations to other i c cells on row 32 where a discrete uniform distribution was applied. When 

Crystal Ball “Assumptions” were defined, Crystal Ball colored each input cell bright green

During a M onte C arlo s imulation, t he M onte Carlo tool ( e.g. Crystal B all) pl aced a 

numerical value in each of the ic cells on row 32. Then the spreadsheet was exercised to perform 

interpolations i n a ll t he D elta w orksheets. T he r esulting e rror s ample va lues f or t he e ntire 

reference NTE event were r eturned t o t he M ethods worksheet Parameters ar ea, and then the 

Methods w orksheet Emission Calculations s ection computes ∆emissions us ing t hree m ethods, 

full model and validation to generate one set of the 14 output values described in the Summary 

section. T he simulation tool stores the set of random input values from row 32 a s well as the 

output va lues i n a n Excel data b ase from w hich t he cor responding s ets of va lues can later b e 

extracted. Once each trial was completed, the simulation tool randomly sampled a second set of 

input values from the respective probability distributions, placed the values in the cells on r ow 

32, exercised the spreadsheet again, stored the input and output values, and went to a third trial, 

etc. Typically 40,000 to 65,000 trials, depending on the reference NTE event, were used in this 

project with this Error Model workbook. 

Note t hat t here ar e t hree w ays t he us er c an control t he ef fect of the ic values in the 

emissions calculations:

Mode control in Summary A6, 

Include / exclude switches in ErrorControl column AD, and 

Specification of i nput r andom va riables ( “Assumptions”) a nd t heir pr obability 

distributions in the Methods worksheet row 32. 

These three ways of controlling the ic values are independent, but the effects are interdependent 

as follows. M ode control determines what categories of errors are added into the calculations. 

Mode controls categories of errors are classified as: 

1. Mode 0 - no errors included 

2. Mode 1 - “all” but ‘environmental’ errors included 

3. Mode 2 - “all” errors added into the calculations 

“All” in this context represents those error surfaces turned on by the switches in the ErrorControl 

worksheet. T he i nput r andom va riable di stribution c ontrols t he di stribution of t he s ampled i c

values applied du ring Monte Carlo simulation for the several D elta e rror surfaces. M ode and 

ErrorControl switches must be appropriately turned on for the effects of the sampled i c values to 

be i ncluded i n t he emissions di fference results. These controls af fect t he calculations i n t he 

Methods worksheet Parameters and Emission Calculations sections.
REPORT 03.14936.12 D-11 

2.5.5 Methods Worksheet: Emission Calculations

In t he area of r ows 6 - 81 of c olumns C H – CP the brake-specific emissions a nd 

∆emissions cal culations ar e pe rformed using t he va riables and parameters ge nerated in the 

Parameters s ection. T hree s ets of col umns, structured similarly, calculate the f ull mode l, 

validation model, time alignment and drift correction for the following methods: 

1. Method 1 calculations are applied in columns CH – CJ, 

2. Method 2 calculations are applied in columns CL – CN, and 

3. Method 3 calculations are in columns CP. 

Columns CH - CJ for Method 1 are typical of the methods where the structure is the same, but 

the formulas are a little different. Column CH performs the PM emission calculations for the 

AVL PEMS, column CI performs for the Horiba PEMS and column CJ for the Sensors PEMS. 

The structure of the three columns is the same. Formulas implemented in the three columns are 

the s ame, but t he equations i mplementing t he f ormulas a pply va riables a nd pa rameters 

appropriate to the respective PEMS. 

As an example of the calculation for PM Method 1 we will examine column CH in detail. 

The full model calculation was accomplished in cells CH48 – CH54. The ideal emissions result 

was br ought i nto t he a rea b y e quation i n CH51. Full m odel P M emissions (ePM) in g/kw-hr 

were calculated in CH54. C ells CH55 – CH59 are information-only diagnostic aids. T he full 

model Method 1 result in CH54 is calculated by the formulas in Figure C-1. 

m PM is a flow weighted particulate matter exhaust concentration in g/mol 

( ) ( ) ( ) ∑

∑

=

=





 



 ⋅ ∆





 



  ∆ 

 



 



 





=⋅ N

i

i i

N

i

PM i

PM rpmSpeed mNT t

t

s

mol

n

mol

g m

ge kW hr

1

1

3600*1000*60

*14159.3*2**

* *

/



FIGURE C-1. BRAKE-SPECIFIC PM BY METHOD 1

In the formula for the full model mode 2, delta error values sampled from the Delta worksheets 

1, 2, 4 a nd 5 ha ve be en a dded i n m PM . S imilarly, de lta e rror va lues s ampled from D elta 

worksheets 20-23, 25, 27 and 28 have been added to the exhaust flow, delta errors sampled from 

worksheets 29-32 and 34-35 were added to torque, and worksheet 43 deltas were added to speed. 

The ∆t values are equal (1 second) and therefore cancel out of the equation. 

The va lidation m odel c alculation was accomplished in the cel ls C H79 – CH81. 

Validation model PM emissions (g/kW-hr) was calculated in cell CH81.
REPORT 03.14936.12 D-12 

Calculations for PM by Method 1 described above for the AVL PEMS in column CH are 

similar f or the H oriba a nd S ensors P EMS by Method 1 i n c olumns C I and C J, r espectively. 

Similar calculations for PM by Method 2 are presented in columns CL – CN, and by Method 3 in 

column CP.

2.6 Worksheet 4: Constants and Equations

The Constants&Eqns tab was strictly a snapshot of equations used in the brake-specific 

emissions c alculations. It di splayed the e quations a nd c onstants i mplemented i n s preadsheet 

formulas of the Methods worksheet 

 

2.7 Worksheet 5: SS PM Error Surface

The 7 Delta C O SS worksheet is a t ypical D elta w orksheet. Its f unctional s tructure, 

formulas, charts and operation are very similar to the following worksheets:

• 20 Delta Exhaust Flow SS 

• 22 Delta Exhaust Flow Pulsation 

• 23 Delta Exhaust Flow Swirl 

• 30 Delta Torque DOE Testing 

• 45 Delta CO2 SS 

•

With minor changes in charts and structure, its function, formulas and operation are also similar 

to the following worksheets: 

• 1 AVL Delta PM SS

• 1 Horiba Delta PM SS

• 1 Sensors Delta PM SS

• 2 AVL Delta PM Transient

• 2 Horiba Delta PM Transient

• 2 Sensors Delta PM Transient

• 4 AVL Delta PM AtmosPressure

• 5 AVL Delta PM Ambient Temp

• 13 Delta NMHC SS

• 14 Delta NMHC Transient 

• 19 Delta Ambient NMHC

• 21 Delta Exhaust Flow Transient 

• 25 Delta Exhaust EMI-RFI

• 29 Delta Dynamic Torque

• 31 Delta Torque Warm-up

• 32 Delta Torque Humidity 

• 34 Delta Torque Interpolation 

• 35 Delta Torque Engine Manuf 

• 42 Delta Fuel Engine Manuf 

• 43 Delta Dynamic Speed

• 44 Delta Dynamic Fuel Rate
REPORT 03.14936.12 D-13 

• 46 Delta CO2 Transient 

The following provides a brief summary of the 7 Delta CO SS worksheet: 

• Rows 1 – 7 contain descriptive information about the error surface implemented in the 

worksheet. 

• Rows 8 - 42 present the error surface in columns A – L. Other columns, M – W, on these 

rows generate a lookup table used with an interpolation routine. 

• Figures A, B and C follow. 

• Rows 76 – 379 calculate t he ∆CO S S e rror va lues for e ach row o f t he reference NTE 

event. These values were returned to the Methods tab Parameters section.

The following paragraphs describe in further detail functions in the 7 Delta CO SS worksheet: 

Data from the error surface (rows 13 – 42, columns A – L, in this Delta worksheet) must 

be e ntered i n s orted or der ( sorted on Lab N ominal c olumn C i n a scending or der) f or pr oper 

operation of t he x -lookup-interpolation f unction. T he three f igures chart t he e rror function. 

Figure A , in similar D elta ta bs, may pl ot several da ta s ets v ersus t he x-value, Lab N ominal 

(column C ). F igure A y-values are C O % Lab N ominal ( column C ), and m ay a lso pl ot 99 th

percentile, 50th percentile (median) and 1st percentile.

Related error surface data are plotted in Figure B. Figure B plots several data sets versus 

the s ame x -value, Lab Nominal ( column C ). F igure B y -values ar e t he di fference, CO % 

(PEMS) – CO % (lab, nom). T he differences plotted may not correspond exactly to the values 

shown i n F igure A be cause of t he s tatistical pr ocedure a pplied i n calculating t he di fferences 

shown in Figure B. Figure B plots the 99th percentile (column I), the 50th percentile (median) 

(column H) and the 1st percentile (column G). In addition to the error surface data, Figure B also 

shows the interpolation line designated ic = xx (column V), and the reference NTE event values 

on the interpolation line (column F rows 80 through end of the reference NTE event versus Lab 

Nominal x-values in column B rows 80 t hrough end of the reference NTE event). W hen i c = 

+1.414319083, the interpolation line plots on the 99th percentile. When i c = 0, the interpolation 

line plots on the 50th percentile. When ic = -1.414319083, the interpolation line plots on the 1st

percentile. The reference NTE event always plots on the interpolation line, with points at the xvalues in the reference NTE event. 

The error surface data were also plotted in the format of Figure C. Again the x-axis was

the same Lab nominal (column C). T his time the y-axis data are the i c values. T hus, the 99th

percentile pl ots at +1.414319083, the 50th percentile plots at 0 a nd the 1st percentile plots at -

1.414319083. T he interpolation line plots at the value of i c, and the reference NTE event plots 

on the interpolation line at the x-values in the reference NTE event. If appropriate value labels 

were di splayed i n F igure C , t he va lues w ould represent t he e rror s urface pl otted on a z -axis 

above the two-dimensional x -y pl ane. These error s urface v alues a re di splayed graphically i n

Figure B.

Now consider inner rows 13 – 41 in the look-up table in columns T – W. Column T is a 

repetition of the x-value from column C. Column U calculates a row-to-row ∆ for the x-values 

in c olumn T f or us e i n i nterpolation. C olumn V c omputes t he i nterpolation l ine l inearly 

interpolated according to the value of i c between the median and the 99th percentile if i c > 0 (on 

median if ic = 0 and on 99th percentile if ic = +1.414319083); and between the median and the 1st
REPORT 03.14936.12 D-14 

percentile if i c < 0 (on median if i c = 0 and on 1st percentile if i c = -1.414319083). Only one i c

value (from cell E80) is applied in this calculation of the interpolation line. The Microsoft Excel 

vertical lookup function VLOOKUP is applied to the table in rows 12 – 42 in columns T – W. 

This is done in rows 80 and down in column F. B ecause of the way the VLOOKUP function 

operates, the first row cells T12 and V12, and the last row cell W42 (all three cells distinguished 

by darker line borders) contain formulas or values different from the formulas of the inner rows. 

The formula in cell T12 assures that the lookup function can always find an x-value in its table. 

The formula in V12 and the value in W12 assure that the interpolation in cells F80 to the end of 

the reference NTE event data returns the nearest ∆CO SS value on the interpolation line if the xvalue is outside the range of the error surface lab nominal values.

Before going t o t he i nterpolation a ccomplished i n F 80 a nd dow n, c onsider br iefly t he 

formulation on rows 12 – 43 in columns O – R. This formulation considers one x-value from the 

reference NTE event, t he f irst one , i n c ell B80 and s elects t he t wo adjacent r ows i n t he e rror 

surface be tween w hich t o i nterpolate on t he B80 x -value. T he r esult i s f ormed on r ow 43 i n 

these columns and then the “check” cell G80 accomplishes the i c controlled interpolation. This 

provides an alternative calculation check on one row in the reference NTE event. 

Now consider the interpolation for each point in the reference NTE event. C olumn B, 

row 80 a nd down, brings the lab nominal x-value from the Methods worksheet reference NTE 

event. For this D elta worksheet, that x -value i s CO % (lab,nom). T he out-of-range flags ar e 

information-only indicating points in the reference NTE event with x-value out of the range of 

the error surface lab nominal. The i c value for this Delta worksheet was brought into cell E80 

from the Methods worksheet ic area. E ach point in the reference NTE event was interpolated 

with the same i c value, but with its own x-value. Recalling that the interpolation line in column 

V was computed with this one i c value, the x-interpolation between the appropriate two adjacent 

rows in the error surface can now be accomplished. This requires using the x-value on each row 

in c olumn B , B 80 a nd dow n, i n the V LOOKUP f unction, a nd pe rforming t he r equired 

calculation using the looked-up values and deltas from the look-up table. The calculation is done 

with the formulas in cell F80 and down. The values computed in column F, cell F80 and down 

through the reference NTE event, could be considered elements of a column matrix or vector, 

and are returned to the Methods worksheet Parameters section.

In t he M onte C arlo s imulation, t he M ethods w orksheet combines t his reference NTE 

event r esult ve ctor f rom t he 7 Delta C O SS worksheet with similar r esults f rom ot her e rror 

surfaces, calculates ∆emissions by three methods, full model and validation to produce a set of 

14 output values (“Forecasts” in Crystal Ball terminology) described in the Summary worksheet

section. This was done having input i c values (including one ic value for this Delta CO SS) all 

chosen b y random s ample f rom t he a ppropriate t runcated no rmal or u niform di stribution as 

explained i n t he M ethods w orksheet section. T hen a nother s ample s et of randomly s ampled 

values was input (only one ic value coming to this Delta function again). T he reference NTE 

event CO SS vector was recomputed with the one new i c value, returned to Methods worksheet 

and another set of 14 output values was produced. This process was repeated many times until a 

statistical conve rgence criterion, described i n S ection 2, w as s atisfied. Typically, 40,000 t o 

65,000 s ets of i nput va lues and 14 out put va lues w ere produced t o s atisfy t he c onvergence 

criterion with this Error Model spreadsheet. 
REPORT 03.14936.12 D-15 

The number of r ows i n t he D elta w orksheet r eference N TE e vent a rea ( rows 80 a nd 

down) s hould m atch t he num ber of r ows i n the reference NTE event applied i n t he M ethods 

worksheet for proper function of Figures B and C. The starter spreadsheet has been set up with 

the r ange of charted reference NTE event s eries e xtending t hrough r ow 379 i n t his D elta 

worksheet. T he ba lance of t he s preadsheet s hould calculate cor rectly w hen a r eference N TE 

event is properly entered in the Methods tab and Parameters formulas properly aligned, although 

figures like B and C will not display properly until the last row of the reference NTE event is 

coincident with the end of the range of the charted reference NTE event series. T his could be

done m anually i n each Delta w orksheet where needed, ho wever, t he m acro was d esigned to 

convert t he fully ‘loaded’ s tarter w orkbook after t he reference NTE event was e ntered i n t he 

Methods worksheet. The macro uses the row count in the reference NTE event, aligns formulas 

in the Methods worksheet Parameters area, and eliminates extra rows in the reference NTE event 

area of each appropriate Delta worksheet. Again, the macro will do the operations correctly only 

on a fully ‘loaded’ starter workbook set up with 300 rows of formulas in the Methods worksheet

Parameter area, and in each of the Delta worksheets using the reference NTE event.
REPORT 03.14936.12

APPENDIX E 

EMS OPERATION LOG
1REPORT 03.14936.12 E-1 

Date PEMS Description Reason Solution

2/25/2008 Horiba

Lost communication with 

TRPM laptop on a 

regular basis

Standard ethernet 

cable does not fit 

properly into this 

laptop

A industrial grade ethernet 

cable was provided by 

Horiba which solved t his 

problem

2/28/2008 Horiba 1

Could not control dilution 

ratio if external flow 

meter is not connected

Unknown Horiba fixed the problem

2/28/2008 Horiba 2

Java software does not 

display the measured 

values that are in the 

Labview software

Unknown Software update fixed the 

problem

3/3/2008 Horiba 2 Dilution air flow not 

stable

Dilution air 

pressure too high

Performed a dilution air 

flow adjustment per 

Horiba

3/4/2008 Sensors 1 Lookup table for MPS2 

repeatedly failed Unknown

Sensors said the criteria is 

too s tringent and it is fine 

as long as it visually looks 

good

3/4/2008 Horiba 1 Dilution Ratio c ontrol is 

still somewhat erratic

Bad PID 

constants

Adjusted PID constants 

to new values suggested 

by Horiba

3/14/2008 Horiba 1

Pressure transducer Pt1 

would not respond to 

calibration

Pressure 

transducer was 

broken

Replaced with new part 

from Horiba

3/21/2008 Horiba 

1,2,&3

Unable to pass Part 1065 

sample flow linearity 

verification

Sample flow is an 

inferred not 

measured value

Linearity verification 

performed on dilution and 

total flow at MASC 

request

3/24/2008 Horiba 

1,2,&3

Sample flow failed 

Horiba check with 

provided external flow 

meter

Flow calibrations 

needed updating

Flow coefficients updated 

for all three units

3/26/2008 Horiba 

1,2,&3

Unable to pass Part 1065 

dilution flow linearity 

verification

Dilution flow 

coefficients 

needed updating

Performed "Dilution Ratio 

Accuracy Adjustment" as 

instructed by Horiba
2REPORT 03.14936.12 E-2 

Date PEMS Description Reason Solution

4/14/2008 Horiba 3 Java software freezes 

when loading

Wrong 

parameters set in 

a config file

AVL programmed an 

offset into their software 

to account for this, laterit 

appeared to be a 

grounding problem in the 

Semtech DS

4/22/2008 Horiba 3

Java softare is not 

communicating with the 

Labview software

Improper 

software 

configuation

Parameters adjusted to fix 

the problem

6/13/2008 Horiba 

2200

Horiba system is unable 

to log the ISO-1576 

ECM broadcast from the 

International Engine

Software did not 

have this 

capability

International engine was 

replaced with a heavyduty Volvo engine that 

uses J1939 broadcast

6/19/2008 Horiba 1

TRPM software not 

reading the same exhaust 

flow as OBS-2200 

software

Calibration 

coefficient is 

wrong

Manually adjust the 

calibration coefficient to 

get the readings to match

6/28/2008 Sensors 1

Sample valve for crystal 

remains in "transient" 

state every time it 

attempts to sample

Stepper motor 

attempting to turn 

too quickly (not 

enough torque)

Sensors readjusted the 

stepper motor speed

7/31/2008 Horiba 2

The OBS-2200 software 

would not trigger the 

OBS-TRPM to start 

sampling during an NTE 

Connector wired 

incorrectly

Reduced the speed of the 

stepper motor

8/8/2008 Horiba 2 Dilution flow is too l ow Replaced with connector 

from second unit

8/12/2008 Horiba 

2200

OBS-2200 software 

unable to read the 

reference torque value 

from J1939 broadcast

Unable to enter 

the proper data 

bit location

Perform dilution flow 

adjustment using internal 

pressure regulator

8/14/2008 Sensors 2 Could not get any of the 

crystals to os cillate

Power supply in 

CQCM head was 

likely burnt out

Enter the reference torque 

value manually

8/15/2008 Horiba 

2200

OBS-2200 laptop would 

not boot up Unknown

Hard drive was placed in 

another identical laptop, 

broken laptop was 

shipped back to Horiba
3REPORT 03.14936.12 E-3 

Date PEMS Description Reason Solution

8/18/2008 Sensors 1 Bypass flow was not 

updating

8/21/2008 Sensors 1

Crystal frequencies, 

corona currents, and 

voltages were dropping 

out

High temperature 

causing 

communication 

problems

Attempted to reduce test 

cell temperature, but 

problem persisted

8/26/2008 AVL 1

Unit switched into Z ero 

Check in the middle of 

the test

Uknown, possibly 

a pressure out of 

limit

Unit switched back into 

sample after approx. 30 

seconds

8/27/2008 Horiba 1 OBS-2200 laptop failed, 

would no longer boot Laptop broken New laptop provided by Horiba

8/27/2008 Horiba 1 OBS-2200 laptop would 

not read J1939 Unknown Sent back to Horiba for repair

8/27/2008 Horiba 1 TPA would not zero

Zero function tied 

to gas analyzers, 

looking for gas 

flow

Use the CAL function 

instead of the ZERO to 

just zero TPA

8/27/2008 Horiba 1

TRPM Java software 

unable to log exhaust 

flow (Labview works)

Error in software Installed new version of 

Java software

8/27/2008 Horiba 1 EAD check w ill fail 

repeatedly

Tolerances are 

too tight for test 

cell operation, 

tolerances 

relaxed.

Ignored tolerances set in 

software for check

8/27/2008 AVL 1

Analog output signal to 

MSS would clip at 2 

mg/m3 range

Soot 

concentration too 

high

Output switched to 0-10 

mg/m3, D R setpoint 

increased from 3 to 6

8/27/2008 AVL 1 DR would oc casionally 

stop controlling

Too much 

moisture in the 

system

Unit purged for moisture 

overnight, and firmware 

upgraded

8/27/2008 Sensors 1

PPMD internal 

temperature was out of 

limit

Test cell 

temperature too 

hot

A new back panel was 

installed w ith two fans to 

promote cooling

8/27/2008 Sensors 1

Bypass flow from TSI 

flowmeter was not 

reading in software

Faulty com cable Replaced TSI cable
4REPORT 03.14936.12 E-4 

Date PEMS Description Reason Solution

8/27/2008 Sensors 1

CQCM communication 

dropped out during 

testing

CQCM power 

supply failed Power supply replaced

8/27/2008 Sensors 1

CQCM communication 

dropped out during 

testing

Unknown

CQCM comm chip 

replaced with a newer 

model

8/27/2008 Sensors 1 Corona needle high 

voltage erratic Unknown Fixing the comm issue resolved this problem

8/27/2008 Sensors 1 PPMD communication 

was dropping out

High internal 

temperature

Directed cooling air at 

PPMD

9/10/2008 Horiba 1

Compressor tripped 

breaker, turned off AVL 

unit

Combining AVL 

and compressor 

on same 15A 

circuit was too 

much current

Horiba compressor 

moved to dedicated 15A 

circuit

9/10/2008 Horiba 1

Make up air flow was 

0.7 lpm instead of 2.3 

lpm

PID constants 

incorrect Modified PID constants

9/12/2008 AVL 1

Analog output signal to 

MSS would s till br iefly 

clip a t 10 m g/m3 range

Soot 

concentration too 

high

Logarithmic analog output 

added as a firmware 

update

9/12/2008 Sensors 1

Negative emissions 

reported even at high 

emission levels on NTE

High dilution, 

crystal saturation 

(no grease), 

crystal 

stabilization

Do not use MPS2, grease 

crystals, wait longer for 

PPMD to warm up

9/18/2008 Sensors 1

The bypass flow would 

increase when crystal 1 

on PPMD1 would 

sample

Crystal may be 

installed 

backwards so 

that it is always 

sampling

Set crystal 1 to be the 

reference crystal

9/18/2008 Sensors 1

Sample flow dr ifting 

during steady state engine 

operation

Temperature 

estimate is based 

on mixing of two 

flows, if the 

estimate is off 

temperature will 

change the flow

Adjusted the parameters 

in the temperature 

estimate
5REPORT 03.14936.12 E-5 

Date PEMS Description Reason Solution

9/18/2008 Sensors 1 No corona current was 

measured for two crystals

Crystals were 

shorted to ground Crystals were replaced

9/18/2008 Sensors 1 High voltage reading was 

low on two crystals

Corona needles 

were too close to 

the crystals

Needles repositioned

9/20/2008 Sensors 1

Sample flow 

measurement was too 

low

Sample flow TC 

was in excess 

flow return, which 

had low flow due 

to connection to 

CVS

TC moved into t he main 

exhaust

9/26/2008 AVL 1 Error: MFD Temperature 

out of Spec

Test cell 

temperature too 

hot

Repos itioned c hiller air to 

blow on the AVL unit

9/30/2008 Sensors 1 Lower than expected 

emissions for the PPMD

Inaccurate sample 

flow temp c ausing 

low sample flow 

meas, corona 

needles not 

positioned 

properly

Repos ition the sample 

flow thermocouple, adjust 

corona needle

10/9/2008 Sensors 1

PPMD sampling de layed 

several seconds after 

trigger

Delay in 

communications

Increased residcence time 

inside PPMD up to 3 

seconds

10/9/2008 Sensors 1 Inaccuracies in flow 

measurement

System uses an 

assumed inlet 

pressure for 

MPS2

Added pressure 

measurement downstream 

of MPS1 to account for 

changes in pressure due 

to MPS2

10/9/2008 Sensors 1

Crystal sampling as soon 

as it becomes available, 

including in an NTE event

Software logic

Software modified so that 

an available crystal waits 

for the start of the next 

NTE event

10/9/2008 Horiba 1

Compressor supplying 

dilution air stops working 

in middle of test (2 units)

Unit shutting off 

due to overheat 

protection

Cool air provided to the 

compressor (temporary 

solution)
6REPORT 03.14936.12 E-6 

Date PEMS Description Reason Solution

10/9/2008 AVL 1

MSS concentration 

reporting too high in 

Sensors software

Sensors not 

correcting 

concentration 

from 0°C to 

20°C (~8%)

Semtech DS software 

updated to include the log 

range and volume 

correction

10/10/2008 Sensors 1 Bypass flow too low

Flow leaking 

through the 

carbon filter 

connection

Tightened the carbon filter

10/15/2008 Sensors 1 PPMD emission results 

were inconsistent

Booker had 

decreased the 

crystal flow from 

0.4 to 0.2 slpm to 

increase loading 

time

Final crystal sample flow 

set at 0.5 slpm

10/15/2008 Sensors 1 PPMD emissions were 

lower than expected Uknown

Crystal sensitivity 

adjusted from 125 hz/μg 

to 100 h z/μg (increasing 

sensitivity by 25% )

10/15/2008 Sensors 1

PPMD wouldn't sample 

when external trigger 

activates on Semtech DS

Unknown

Powered off hardware 

and laptop and restarted, 

problem was resolved

10/16/2008 Sensors 1 PPMD unable to 

communicate with DS Unknown

Triggered the PPMD 

manually through Host 

software (unofficial 

testing)

10/28/2008 Horiba 1 Dilution flow inaccurate Unknown Recalibrated VFM

11/3/2008 Sensors 1 PPMD block pressure 

low

Uknown, block 

pressures were 

activated in the 

software

Aborted cycle, powered 

down the PPMD and 

restarted, problem was 

fixed

11/4/2008 Sensors 

1

Multiple crystals stopped 

oscillating, when one was 

enabled another would 

disable

Grease loading 

slightly too high?

Re cleaned and greased 

crystals

11/4/2008 Horiba 1

Unable to maintain 

setpoint for total flow 

near end of the cycle

High filter loading

Adjusted the valve on the 

total flow pump to allow 

more flow
7REPORT 03.14936.12 E-7 

Date PEMS Description Reason Solution

11/5/2008 Horiba 1 Sample flow accuracy is 

out of spec

Caused by 

inaccurate dilution 

flow measurement 

(cause of this 

uknown)

Recalibrated VFM

11/6/2008 Horiba 1 Dilution flow is still 

inaccurate Unknown Recalibrated VFM

11/7/2008 Sensors 1

MPS Dilution Flow 

Major audit could not 

pass

Overheated None, cycle voided

11/7/2008 Horiba 1 External compressor 

stopped working Overheated None, cycle voided

11/7/2008 Horiba 1 External compressor 

stopped working Overheated

None, cycle voided, 

installed 2nd compressor 

in pa rallel

11/12/2008 Sensors 1

PPMD results not 

included in the DS results 

file

Descrepancies in 

the time stamps Never resolved

11/12/2008 Horiba 1 Dilution compressor still 

stoppi ng sometimes Overheating

Connected two 

compressors in pa rallel, 

so the 2nd will run if the 

first stops

11/12/2008 Sensors 1

Slope on flow for daily 

audits sometimes fails to 

0.97 or 1.03

Monthly 

tolerances too 

tight for da ily 

checks

Tolerances relaxed slightly 

for daily checks to save 

time

11/12/2008 Horiba 1

Sample flow check is 

sometimes not within 

tolerance

Monthly 

tolerances too 

tight for da ily 

checks

Tolerances relaxed slightly 

for daily checks to save 

time

11/24/2008 Horiba 1 Sample flow accuracy is 

out of spec

Inaccurate 

dilution flow 

measurement

Recalibrated VFM

11/25/2008 AVL 1 MSS failed the external 

DR audit repeatedly

Unknown - all 

internal checks 

passed

Requirement of 

performing external audit 

on da ily ba sis was 

removed
8REPORT 03.14936.12 E-8 

Date PEMS Description Reason Solution

12/1/2008 Sensors 1 EFM reading inaccurate 

at the end of each test

Solenoid not 

switching 

properly during 

the 1 hour 

autozero, EFM 

zeroing while 

open to flow

Solenoid fixed, auto zero 

disabled?

12/1/2008 Horiba 1

Newest compressor that 

was shipped still stopping 

sometimes

Overheating

Continued to use two 

compressors in parallel, a 

new compressor was 

shipped from Horiba

12/1/2008 Horiba 1

Post processor file size 

too big for Excel (>200 

MB)

Steady state data 

processed at 1Hz 

instead of 10Hz

Continued to use two 

compressors in parallel

12/11/2008 Horiba 1 DCS signal flat lined 

during testing Uknown

Problem did not occur 

again when system was 

restarted

12/15/2008 Horiba 1 TRPM data filed was not 

saved by Java software Unknown None, cycle was void

12/16/2008 Horiba 1 External compressor 

stopped working Overheated None, cycle voided

12/18/2008 Horiba 1 External compressor 

stopped working Overheated None, cycle voided

1/14/2009 Horiba 1 Sample flow accuracy is 

out of spec

Caused by 

inaccurate dilution 

flow measurement 

(cause of this 

uknown)

Recalibrated VFM

1/14/2009 Horiba 1 Dilution flow inaccurate Unknown Recalibrated VFM

1/26/2009 Sensors 2

Semtech DS lost 

communication with 

laptop

Unknown

Couldn't monitor data, but 

data was still recorded on 

compact flash card

1/28/2009 Horiba 2

TPA switched from 

measure to standby at the 

start of the test

Software glitch? Did not occur again
9REPORT 03.14936.12 E-9 

Date PEMS Description Reason Solution

1/28/2009 Horiba 2

TRPM would switch out 

of filter sample mode 

once 30 seconds had 

elapsed

Faulty software 

logic

Software modified so that 

filter will never switch out 

of sample mode while in 

an NTE event

1/28/2009 Horiba 2

Filter sampling was 

beginning 10 s ec after 

start of NTE instead of 5

OBS-2200 is 

delayed 5 

seconds in its 

response

Horiba chose not to make 

any changes

1/30/2009 AVL 2 Error: No di lution air 

available

On-board pump 

was leaking

Pump was replaced with 

new part shipped from 

AVL

2/2/2009 Sensors 2

Semtech DS lost 

communication with 

laptop, could not 

reconnect

Unknown

Powered down unit, 

recovered data the next 

day

2/5/2009 Sensors 2

Semtech DS was losing 

communication with 

laptop

Unknown

Couldn't monitor data, but 

data was still recorded on 

compact flash card

2/18/2009 Horiba 2 External compressor 

stopped working Overheated None, cycle voided

3/2/2009 Horiba 3

Software zero pressure 

transducer function would 

not work after repeated 

attempts

Unknown

Shut dow n equipment 

attempted it again several 

hours later and it worked

3/3/2009 Horiba 3 External compressor 

stopped working Overheated None, cycle voided

3/5/2009 Sensors 3

Semtech DS lost 

communication with 

laptop

Unknown

Couldn't monitor data, but 

data was still recorded on 

compact flash card

3/6/2009 Horiba 3 External compressor 

stopped working Overheated None, cycle voided

3/23/2009 Horiba 

2200

OBS-2200 software 

locked up during test Unknown

Software worked when 

rebooted, but data for the 

cycle was lost

3/24/2009 Sensors 

DS

Couldn't connect to the 

Semtech DS repeatedly

LAN circuit 

board was likely 

damaged

Unit sent back to Sensors 

for repair
10REPORT 03.14936.12 E-10 

Date PEMS Description Reason Solution

3/25/2009 Sensors 3

PPMD wouldn't sample 

when NTE trigger 

activates on the Semtech 

DS

Unknown Turning the unit off and 

back on fixed the problem

4/3/2009 AVL 1

There is a voltage offset 

between what the AVL 

unit outputs and the 

Semtech DS reads

Unknown Switched constants

4/13/2009 AVL 2

Power inverter shut dow n 

during radiated immunity 

test

EMI/RFI Test Replaced bypass pump

4/15/2009 Sensors 2

Communication with the 

laptop dropped out 

repeatedly during bulk 

current injection

EMI/RFI Test Head sent back for repair

4/20/2009 Sensors 2

Communication with the 

laptop dropped out 

during radiated immunity

EMI/RFI Test Replaced with bracket 

from other unit

4/21/2009 AVL 2

Power inverter shut dow n 

during conducted 

transient test

EMI/RFI Test No corrective action 

taken

4/21/2009 AVL 2

Power inverter shut 

down, then start smoking 

during conducted 

transient

EMI/RFI Test

Unit was shipped back to 

AVL, replaced with other 

unit, problem was not 

observed again

4/21/2009 Sensors 2

PPMD shut dow n 

completely several times 

during conducted 

transient tests

EMI/RFI Test No corrective action 

taken

4/24/2009 Horiba 3 Flow was erratic during 

bulk current injection test EMI/RFI Test No corrective action taken

4/27/2009 Horiba 3

Exhaust flow was reading 

very high during bulk 

current injection test

EMI/RFI Test No corrective action 

taken

4/28/2009 Horiba 3

Lost communication with 

laptop during radiated 

immunity test

EMI/RFI Test

Inverter replaced with 

backup, test was not 

repeated
11REPORT 03.14936.12 E-11 

Date PEMS Description Reason Solution

4/28/2009 Horiba 3

Exhaust flow was reading 

very high during radiated 

immunity test

EMI/RFI Test No corrective action 

taken

5/15/2009 AVL 3

AVL PEMS blew a fuse 

when trying to pow er up 

after 1st pressure test

Uknown Replaced with AVL 2

7/15/2009 Sensors 3

PPMD reported a 

barometric pressure 

increase when altitude 

chamber was at vacuum

Constants in the 

software were 

backwards for 

the barometric 

pressure

No corrective action 

taken

7/15/2009 Sensors 3

PPMD could not 

maintain a bypass flow of 

4 slpm

Bypass pump 

was dying

No corrective action 

taken

7/21/2009 AVL 2

Soot concentration 

reading erratic during 

vibration (all or ientations)

Vibration Rebooted hardware to 

reconnect

7/22/2009 Horiba 3

L-bracket holding filter in 

ME box broke during 

vibration testing

Vibration

Did not continue with 

more extreme tests to 

prevent damage

7/22/2009 Horiba 3

Total Pi pressure 

transducer would not 

read correctly after 

problem with L-bracket 

occurred

Vibration Test Pressure transducer was 

replaced

7/24/2009 Sensors 2

PPMD could not 

maintain total flow due to 

moisture traps opening

Vibration Test No corrective action 

taken