Test Plan to Determine PEMS 

Measurement Allowance for the PM 

Emissions Regulated under the 

Manufacturer-Run Heavy-Duty Diesel 

Engine in-use Testing Program 
Test Plan to Determine PEMS 


Measurement Allowance for the PM 


Emissions Regulated under the 


Manufacturer-Run Heavy-Duty Diesel 


Engine in-use Testing Program 


Assessment and Standards Division 

Office of Transportation and Air Quality 

U.S. Environmental Protection Agency 

and 

California Air Resources Board 

and 

Engine Manufactures Association 

EPA-420-B-10-901 

August 2010 
EXECUTIVE SUMMARY 

This test plan sets forth the agreed upon processes and methodologies to be utilized to 

develop additive, brake-specific, data-driven measurement allowance for PM emissions 

measured by PEMS as required under the HDIUT regulatory program. 

As detailed in this test plan, there is a clear consensus on what components of 

measurement error are intended to be covered by the measurement allowance. Namely, 

the allowance is to be calculated in a manner that subtracts lab error from PEMS error. 

Specifically, utilizing Part 1065 compliant emissions measurement systems and 

procedures for both the lab and PEMS, the lab error associated with measuring heavyduty engine emissions at stabilized steady-state test points within the NTE zone, will be 

subtracted from the PEMS error associated with measuring heavy-duty engine emissions 

utilizing PEMS over events under a broad range of environmental conditions. This 

subtraction will yield “PEMS minus laboratory” measurement allowance. The 

experimental methods and procedures specified in this test plan for determining, 

modeling, and comparing each of the various components of measurement error are 

designed to generate statistically robust data-driven measurement allowance for the PM 

emissions. 

Successful completion of this test plan is part of the resolution of a 2001 suit filed against 

EPA by EMA and a number of individual engine manufacturers. The suit challenged, 

among other things, certain supplemental emission requirements referred to as “not-toexceed” (NTE) standards. On June 3, 2003, the parties finalized a settlement of their 

disputes pertaining to the NTE standards. The parties agreed upon a detailed outline for a 

future regulation that would require a manufacturer-run heavy-duty in-use NTE testing 

(“HDIUT”) program for diesel-fueled engines and vehicles. One section of the outline 

stated: 

“The NTE Threshold will be the NTE standard, including the margins built into the 

existing regulations, plus additional margin to account for in-use measurement accuracy. 

This additional margin shall be determined by the measurement processes and 

methodologies to be developed and approved by EPA/CARB/EMA. This margin will be 

structured to encourage instrument manufacturers to develop more and more accurate 

instruments in the future.” 

Given the foregoing, the work to be completed under this test plan is a vital component to 

the fulfillment of the settlement agreement, and it is vital to the successful 

implementation of a fully-enforceable HDIUT program. Because of this significance, it 

is critically important that the work detailed in this test plan be carried out in as thorough, 

careful and timely a manner as possible. 

2
 
TABLE OF CONTENTS 

Executive Summary............................................................................................................ 2 

Table of Contents................................................................................................................ 3 

List of Figures..................................................................................................................... 5 

List of Tables ...................................................................................................................... 5 

1 Introduction................................................................................................................. 6 

2 Monte Carlo Error Model and Measurement Allowance ........................................... 8 

2.1 Objective............................................................................................................. 8 

2.2 Background......................................................................................................... 9 

2.3 Methods and Materials...................................................................................... 15 

2.4 Simulation Procedure........................................................................................ 16 

2.4.1 Construction of the Error Surface ............................................................. 16 

2.5 Model Considerations ....................................................................................... 24 

2.5.1 Convergence ............................................................................................. 24 

2.6 Simulation Output............................................................................................. 24 

2.6.1 Sensitivity Variation Effect....................................................................... 24 

2.6.2 Sensitivity Bias Effect............................................................................... 25 

3 Engine Dynamometer Laboratory Tests ................................................................... 26 

3.1 Preliminary Audits............................................................................................ 26 

3.1.1 Objective ................................................................................................... 26 

3.1.2 Background............................................................................................... 26 

3.1.3 On-site meeting to establish 1065 compliance requirements ................... 26 

3.1.4 Methods and Materials.............................................................................. 26 

3.1.5 Data Analysis............................................................................................ 27 

3.1.6 PEMS Manufacturer PM PEMS Commissioning..................................... 27 

3.2 Bias and Precision Errors under steady state engine operation ........................ 27 

3.2.1 Objective ................................................................................................... 27 

3.2.2 Background............................................................................................... 28 

3.2.3 Methods and Materials.............................................................................. 28 

3.2.4 Data Analysis............................................................................................ 30 

3.3 Precision Errors under transient engine operation (dynamic response)............ 31 

3.3.1 Objective ................................................................................................... 31 

3.3.2 Background............................................................................................... 31 

3.3.3 Methods and Materials.............................................................................. 31 

3.3.4 Data Analysis............................................................................................ 35 

3.4 ECM Torque and BSFC.................................................................................... 37 

3.4.1 Objective ................................................................................................... 37 

3.4.2 Data Analysis............................................................................................ 37 

4 Environmental Chamber ........................................................................................... 37 

4.1 Data Analysis for Environmental Tests ............................................................ 38 

4.2 PM Generator Commissioning ......................................................................... 39 

4.3 Baseline............................................................................................................. 39 

4.3.1 Objective................................................................................................... 39 

4.3.2 Background............................................................................................... 39 

3
 
4.3.3 Methods and Materials.............................................................................. 40 

4.3.4 Data Analysis............................................................................................ 40 

4.4 Electromagnetic Radiation................................................................................ 40 

4.4.1 Objective ................................................................................................... 40 

4.4.2 Methods and Materials.............................................................................. 40 

4.5 Atmospheric Pressure ....................................................................................... 41 

4.5.1 Objective ................................................................................................... 41 

4.5.2 Background............................................................................................... 41 

4.5.3 Methods and Materials.............................................................................. 41 

4.5.4 Data Analysis............................................................................................ 43 

4.6 Ambient Temperature and Humidity................................................................ 44 

4.6.1 Objective ................................................................................................... 44 

4.6.2 Background............................................................................................... 44 

4.6.3 Methods and Materials.............................................................................. 45 

4.6.4 Data Analysis............................................................................................ 46 

4.7 Orientation and Vibration ................................................................................. 46 

4.7.1 Objective ................................................................................................... 46 

5 SwRI CVS and CE-CERT Trailer Correlation ......................................................... 47 

5.1 Method and Materials ....................................................................................... 47 

6 Model Validation and Measurement Allowance Determination .............................. 48 

6.1 Model validation ............................................................................................... 48 

6.1.1 Objective ................................................................................................... 48 

6.2 Measurement Allowance Determination .......................................................... 49 

6.2.1 Objective ................................................................................................... 49 

6.2.2 Background............................................................................................... 49 

6.2.3 Methods and Materials.............................................................................. 49 

6.2.4 Data Analysis............................................................................................ 49 

7 Time and Cost........................................................................................................... 50 

7.1 Timeline ............................................................................................................ 50 

7.2 Cost ................................................................................................................... 51 

8 Abbreviations used in Brake Specific Equations...................................................... 51 

4
 
LIST OF FIGURES 

Figure 1. Probability Density Functions For Sampling Error Surfaces............................. 9 

Figure 2. Brake-Specific PM Emissions Calculation For Method 1 ............................... 11 

Figure 3. Brake-Specific PM Emissions Calculation For Method 2 ............................... 12 

Figure 4. Brake-Specific PM Emissions Calculation For Method 3 ............................... 13 

Figure 5. Error Surface: PEMS Vs. LAB ....................................................................... 17 

Figure 6. Error Surface: (PEMS-LAB) Vs. LAB ........................................................... 18 

Figure 7. Error Surface: Final Version ........................................................................... 19 

Figure 8. Overview of Monte Carlo Simulation ............................................................... 22 

Figure 9. Example of A NTE Cycle.................................................................................. 35 

Figure 10. Pressure Histogram.......................................................................................... 41 

Figure 11. Pressure-Time Environmental Test Cycle....................................................... 43 

Figure 12. temperature histogram..................................................................................... 44 

Figure 13. Time Series Chart of Ambient Temperature Test ........................................... 46 

LIST OF TABLES 

Table 1. Allowed Modifications ......................................................................................... 6 

Table 2. Example Of Selection Of Measurement Allowance At 0.02 g/hp-hr NTE 

Threshold .................................................................................................................. 15 

Table 3. Error Surfaces for the BSPM Simulation ........................................................... 23 

Table 4. Engine, Exhaust Configuration, And Steady-State Modes................................. 28 

Table 5. Example Of SS Error Surface............................................................................. 30 

Table 6. NTE Transient Cycle .......................................................................................... 32 

Table 7. Dynamic response inter-NTE Events ................................................................. 34 

Table 8. Concentration and Dilution Ratio Schedule with PM Generator ....................... 38 

Table 9. Atmospheric Pressure Test Sequence ................................................................. 42 

Table 10. Ambient Temperature test sequence................................................................. 45 

Table 11. Projected PM-PEMS Timeline ......................................................................... 50 

Table 12. projected cost estimate...................................................................................... 51 

5
 
 

1 INTRODUCTION 

This test plan will establish a PEMS measurement allowances for PM, as regulated by the 

manufacturer-run on-highway heavy-duty diesel engine in-use test program. The 

measurement allowance will be established using various laboratory facilities and PEMS. 

The measurement allowance will be established in units of brake-specific emissions 

(g/hp-hr), and it will be added to the final NTE PM standard, after all the other additive 

and multiplicative allowances have been applied. This test plan will establish the PM 

measurement allowance. 

The PEMS used in this test plan must be standard in-production makes and models that 

are for sale as commercially available PEMS. In addition, PEMS and any support 

equipment must pass a “red-face” test with respect to being consistent with acceptable 

practices for in-use testing. For example, the equipment must meet all safety and 

transportation regulations for use on-board heavy-duty vehicles. 

Even though the PEMS can not be “prototypes” nor their software “beta” versions, the 

steering committee has already agreed that after delivery of PEMS to the contractor, there 

may be a few circumstances in which PEMS modifications might be allowed, but these 

modifications must meet certain deadlines, plus they are subject to approval by the 

steering committee. Also, any implementation of such approved modifications will not 

be allowed to delay the test plan, unless the steering committee specifically approves 

such a delay. Table 1 summarizes these allowable modifications and their respective 

deadlines: 

TABLE 1. ALLOWED MODIFICATIONS 

Allowed Modifications Before start of... 

Steering committee approved hardware and software modifications 

that affect emissions results; including but not limited to fittings, 

components, calibrations, compensation algorithms, sampling rates, 

recording rates, etc. 

Steady-State 

Testing 

Steering committee approved hardware modifications for DOT 

approval or any other safety requirement approval 

Environmental 

Chamber Testing 

Delivery of any environmental / weather enclosure to contractor Environmental 

Chamber Testing 

Post-processing software to determine NTE results Model Validation 

DOT approval and documentation Model Validation 

Steering committee approved hardware or software that improves 

the contractor’s efficiency to conduct testing and data reduction Always Allowed 

The steering committee approved three different PEMS that includes the AVL MicroSoot Sensor (MSS), the Horiba Transient Particulate Matter (TRPM), and the Sensors 

Proportional Particulate Matter Diluter (PPMD). However, because of the different 

measurement technologies employed by each of these systems, the three different PEMS 

hold slightly different status with respect to determining the PM measurement allowance. 

6 
Because inertial microbalances are already approved for PEMS applications in 40 CFR 

Part 1065, the Sensors PPMD will be one of the PEMS used to determine the 

measurement allowance. And because EPA’s PM standard is based upon a gravimetric 

filter analysis, the Horiba TRPM will also be used to determine the measurement 

allowance. The lowest measurement allowance value between the two will be selected as 

the final measurement allowance for PM. If that value does not validate, then the lowest 

validated value will be chosen. If the lowest validated value chosen is within 0.0075 

g/hp-hr from the lowest non-validated value, then the lowest validated value will be the 

measurement allowance. Otherwise, the MASC will spend up to a $100,000 to figure out 

a resolution to the problem by generating more data or changing the way the validation 

was performed. If that does not lead to a resolution, then Executive Management of EMA 

and EPA will have to settle the issue. 

Note that at the conclusion of successful testing of the Horiba system in this measurement 

allowance program, EPA intends to approve the Horiba system as an alternative for use, 

or EPA may elect to amend 40 CFR Parts 86 and/or 1065 to allow the use of the Horiba 

TRPM or other PEMS that operate upon similar measurement principles. Because the 

AVL system measures only the soot component of PM, the measurement allowance will 

not be determined using the AVL results, unless both the Sensors and Horiba systems fail 

to complete the measurement allowance program. Note that the steering committee may 

determine at the conclusion of the program that the AVL MSS is a viable alternative for 

demonstrating compliance. Under such a circumstance EPA may amend the Heavy-Duty 

In-Use regulation to allow for its use. 

This test plan describes a computer model, a series of experiments that are used to 

calibrate the model, and another series of experiments that are used to validate the 

calibrated model. 

The test plan first describes the computer model. The computer model statistically 

combines many sources of PEMS and lab error, which are nearly impossible to capture 

simultaneously in a single test. The model will use statistics to apply the errors in a way 

that simulates actual running of a PEMS in-use. The model will also consider only the 

portion of error that is attributable to PEMS, and it will subtract the error that is already 

tolerated in an emissions lab today. The model will also calculate and validate results 

according to 40 CFR Part 1065. 

The test plan then describes the series of experiments. These tests will characterize the 

many sources of PEMS and lab error so that the specific nature of the errors can be 

programmed into the computer model. The nature of the error has to do with the way 

PEMS and the lab react to certain conditions. For example, under varying environmental 

conditions such as temperature or vibration, a PEMS might exhibit signal drift, or it may 

record noise that is not a part of the true emissions. 

Next, the experimental results will be entered into the computer model, and the 

measurement allowances are calculated by the model. The model uses a “reference" 

PEMS data set, which will have many “reference NTE events.” The model statistically 

7
 
applies all the errors to the reference data set, calculates results, and saves the results. 

Then the model will be run with all errors set to zero to calculate the ideal results of the 

reference data set. Each difference between a reference NTE event’s result with errors 

and its respective ideal result will be a brake-specific difference that is recorded for later 

use. Then the process repeats using the same reference data set, to which new, 

statistically selected errors are applied, and thus another unique set of differences is 

calculated. As the model continues to iterate and generate more and more results, 

patterns are expected to appear in the output data. These patterns should be the 

distributions of differences, based upon the error that was statistically and repeatedly 

applied to the reference data set. Many difference distributions will be determined: for 

each reference NTE event, for each of the two brake-specific calculation methods (three 

in case of the AVL system only), and for each PEMS. It has been agreed that the 95th 

percentile values of these distributions will be taken as reasonable “worst case” results 

for each reference NTE event. Details on how all these distributions will be reduced to 

determine the PM measurement allowance is given in the “Error Model” section of this 

test plan. 

Because the calculation based on Method 2 and Method 3 require gas-based fuel flow 

calculation based on the measurement of CO2, CO, and NMHC, a decision was made to 

use the gaseous PEMS data for this purpose, without the need to perform gaseous 

measurement during the PM-PEMS program. 

Finally, the test plan describes how the computer model will be validated against realworld over-the-road in-use PEMS operation as well as additional lab testing. For the 

over-the-road testing, PEMS emissions measurements will be conducted, while at the 

same time a reference laboratory will be towed along to measure the same emissions. For 

the lab testing, an attempt will be made to simulate real-world engine operation to 

“replay” an over-the-road test in the lab. Data from these final experiments will be used 

to validate the model, which must be done in order to gain sufficient confidence that the 

model did not establish unreasonable measurement allowances. 

The following sections of this test plan are written as instructions to the contractor or 

contractors who will complete the test plan. 

2 MONTE CARLO ERROR MODEL AND MEASUREMENT ALLOWANCE 

2.1 Objective 

Use Monte Carlo (e.g. random sampling) techniques in an error model to simulate the 

combined effects of all the agreed-upon sources of PEMS error incremental to lab error. 

Create error “surfaces” for the Monte Carlo simulation to sample, based upon results 

from the experiments described in Sections 3 and 4. Exercise the model over a wide 

range of NTE events, based on a single, reference data set of at least 150 but no more 

than 200 unique NTE events. Determine the pollutant-specific brake-specific additive 

measurement allowance for PM. 

8
 
2.2 Background 

The error model uses Monte Carlo techniques to sample error values from “error 

surfaces” that are generated from the results of each of the experiments described in 

Section 3 on engine dynamometer laboratory tests and Section 4 on environmental 

chamber tests. The lab test error surfaces cover the domain of error versus the magnitude 

of the signal to which the error is to be applied (i.e. 1st to 99th percentile error vs. 

concentration, flow, torque, etc.). This is illustrated later in this section. The 

environmental test error surfaces for shock & vibration and electromagnetic & radio 

frequency interference (EMI/RFI) cover the same domain as the lab tests. The 

environmental test error surfaces for pressure and temperature are characteristically 

different because they cover the domain of environmental test cycle time versus the 

magnitude of the signal to which the error is to be applied (i.e. error at a selected time vs. 

concentration). Details on how each surface is generated are given in each of the 

respective sections. These surfaces will already be adjusted to represent PEMS error 

incremental to lab error; therefore, these surfaces are sampled directly by the model. 

The error model will use two different probability density functions (PDFs) as shown in 

Figure 1 to sample the error surfaces, depending upon which experiment the surface 

represents. To sample error surfaces that are generated from all the laboratory test results 

(Section 3), and the environmental test results for shock & vibration (Section 4), the 

model will use a truncated normal PDF because these tests are designed to evenly cover 

the full, but finite, range of engine operation and ambient conditions. To sample error 

surfaces that are generated from the pressure and temperature environmental test results 

(Section 4), the model will use a uniform PDF because these tests are already designed to 

cover the typical range and frequency of the respective conditions. 

Probability Density Functions for Sampling Error Surfaces Once Per NTE Event 

-1.00 

-0.75 

-0.50 

-0.25 

0.00 

0.25 

0.50 

0.75 

1.00 

5 4 3 2 1 0

Relative Probability 

i c 

Lab Tests, Normal, SD=0.60795, truncate @ -1 & 1 

Environmental Tests, Uniform 

Note: A non-truncated normal 

distribution with SD=0.60795 has P 

values of 0.01 and 0.99 at i c =-1 and 

ic=+1, respectively. 

FIGURE 1. PROBABILITY DENSITY FUNCTIONS FOR SAMPLING ERROR 

SURFACES 

9 
The random values that are obtained from both distributions are labeled ic in Figure 1 and 

range from -1 to 1. Note that for the pressure and temperature environmental tests, a 

uniform PDF will be used to sample test time, from which the nearest (in time) calculated 

errors are used. The errors from the other tests will be aligned with the truncated normal 

PDF such that each of the 50th percentile values at each of the tested signal magnitudes is 

centered at the median of the PDF (ic = 0), and the 1st and 99th percentile error values at 

each of the tested signal magnitudes will be aligned with the extreme negative (ic = -1) 

and positive (ic = +1) edges of the PDF, respectively. 

Each error surface will be sampled along its ic axis (y-axis) once per reference NTE event 

trial, and it will be sampled along its parameter value axis (x-axis, e.g., concentration 

(only for AVL MSS), flow, torque, etc...) once per second, within a given reference NTE 

event trial. An error will be determined for a given second and parameter along the error 

axis (z-axis) at the intersection of an ic value and a parameter value. 

To ensure that the magnitudes of the error surfaces are appropriate, each data point used 

to generate the surfaces will be a mean or a weighted mean of 30 seconds of sampling. 

Interpolation will be performed by first linearly interpolating error values at each tested 

magnitude along the selected line perpendicular to the ic axis. Then from that line of 

errors, individual error values will be linearly interpolated at each second-by-second 

signal magnitude of the given NTE event in the reference data set. 

The reference data set to which all errors will be applied will be a large data set of engine 

operation over a wide range of NTE events. This reference data set will be initially 

generated from collections of real-world PEMS data sets. The reference data set should 

contain at least 150 but no more than 200 unique NTE events. Parameters in the 

reference data set may be scaled in order to exercise the model through a more 

appropriate range of parameters (i.e. concentrations, flows, ambient conditions, etc.). If 

the parameters are scaled, care should be taken to maintain the dynamic characteristics of 

the reference data set. 

After the errors are applied, NTE brake-specific PM emissions results are calculated, 

using each of the three agreed-upon NTE calculation methods. The three different brakespecific emission calculation methods for PM referred to in this test plan are i) TorqueSpeed method, ii) BSFC method, and iii) ECM-Fuel Specific method, and these are 

illustrated in Figure 2, Figure 3, and Figure 4, respectively. 

10
 
For all PM PEMS: 

m is a flow weighted particulate matter exhaust concentration in g/mol PM

 g  N   mol 

mPM  * ni * t  mol i1   s   e g / kW  hr  PM N Speedirpm*TiN  m* 2*3.14159* t 

i1 



 60*1000*3600 

Where for AVL: 

m is computed numerically as follows, PM

N   g   mol   mPMi  ni  *t  g  i1   mol   s   mPM    N  mol    mol    n *t  i   

i1   s  

FIGURE 2. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR 

METHOD 1
 

11
 
 

For all PM PEMS: 

m is a flow weighted particulate matter exhaust concentration in g/mol PM

 g  N  mol 

mPM  *n

i 

*t

 mol i1   s   ePM  g / kWhr         mol 6 2 

N n * xTHC   ppm  xCO   % xCO % *10 * 

i  

 i *10  i 2   

 t

M 

C  s  * i



wfuel   g  i1    m  fueli    s         i rpm * i  *2*3.14159 Speed   T N m       60*1000*3600     

Where for AVL: 

m is computed numerically as follows, PM

N   g   mol    mPMi  ni  *t    g  i1   mol   s   mPM    N  mol    mol   ni  *t

i1   s  

FIGURE 3. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR 

METHOD 2 

12
 
For AVL Only: 

ePM  g / kW   hr

   g 

m N  ifuel      g wfuel s

mPM  * *   *t 6 2  mol  MC i



xTHCi ppm *10   xCOi %  xCO2 % *10 

 1        i

  

 Speed    *T N m 

N

 i rpm i *2*3.14159* t

i1 60*1000*3600  

Where: 

   g 

N  mPMi  g / mol*m w ifu  el 

fuel s *   *t

M 6 2

C i1  xTHC   ppm *10  xCO   %  xCO % *10 i i i 2   

 g  

  mPM      mol    g 

m w N  ifuel   

fuel   s * *t

M 6 2

C i1  xTHCi   ppm *10  xCOi   %  xCO2   % *10   i

  

FIGURE 4. BRAKE-SPECIFIC PM EMISSIONS CALCULATION FOR 

METHOD 3 

Next, the NTE events are calculated by each of the three calculation methods, but with no 

error sampled or applied to the reference data set. These results are considered the 

“ideal” results of the reference NTE events. These ideal results are subtracted from each 

respective NTE event result ‘with errors’, and the difference is recorded. Then a new set 

of errors are sampled and applied to the reference NTE event, and the NTE results ‘with 

errors’ are calculated again. The ideal results are again subtracted, and the difference is 

recorded. This is repeated thousands of times so that the model converges upon 

distributions of brake-specific differences for each of the original NTE events in the 

reference data set. 

Then the 95th percentile difference value is determined for each NTE event distribution of 

brake-specific differences for PM for each calculation method. At this point there is one 

distribution of 95th percentile differences for PM, where all the NTE events are pooled by 

the PM emissions for each of the three different calculation methods. Each of the 95th 

percentile distributions represents a range of possible measurement allowance values. 

13
 
 

 

From each of these three distributions of possible measurement allowance values, one 

measurement allowance per distribution must be determined. First the correlation 

between 95th percentile differences versus the ideal PM emission is tested. For each 

calculation method, if a least squares linear regression of 95th percentile differences 

versus ideal PM emissions has an r2 (squared correlation coefficient) > 0.85 and an SEE 

(standard error of the estimate or root-mean-squared-error) < 5 % of the median ideal PM 

emission, then that linear regression equation will be used to determine the measurement 

allowance for that calculation method at the following NTE threshold: 

PM = 0.02 g/hp-hr and 0.03 g/hp-hr 

In cases where extrapolation is required to determine the measurement allowance at the 

NTE threshold, the measurement allowance will be determined using the linear 

regression, but evaluated at the ideal PM emission that is closest to the NTE threshold, 

not extrapolated to the NTE threshold itself. If the linear regression does not pass the 

aforementioned r2

 and SEE criteria, then the median value of the 95th percentile 

differences is used as the single measurement allowance for that calculation method. 

Next, the calculation method is selected. The above procedure will provide three 

measurement allowances, where applicable, one for each of the three different calculation 

methods. To make them comparable, the three measurement allowance values will be 

normalized by the PM threshold and expressed as a percent. Also, if any measurement 

allowance is determined to have a value less than zero, then that measurement allowance 

will be set equal to zero. The calculation method with the minimum normalized PM 

value will be chosen and the corresponding normalized PM value will be selected as the 

best measurement allowance for PM, assuming it validates. If it does not validate, then 

the minimum value that validates will be chosen as long as it is within 0.0075 g/hp-hr 

from the minimum value that did not validate. If the difference between the minimum 

value that validates and the minimum value that did not validate is greater than 0.0075 

g/hp-hr, additional investigation with up to a $100,000 will be spent in order to 

understand why the minimum value chosen did not validate. If the problem is not 

resolved after spending the $100,000, then the matter will be referred to executive 

management of EPA and EMA to decide on the PM measurement allowance. 

Error! Reference source not found. below illustrates the selection of the calculation 

method. The example is based on a hypothetical set of normalized PM measurements for 

the three calculation methods. The minimum of these normalized allowances is used to 

select the best method (highlighted in blue). In this hypothetical case, the BSFC method 

would be selected. 

14
 
 

TABLE 2. EXAMPLE OF SELECTION OF MEASUREMENT ALLOWANCE AT 

0.02 G/HP-HR NTE THRESHOLD 

Allowance at Respective NTE Threshold (%) 

Calc. Method ==> Torque-Speed BSFC ECM fuel specific 

BSPM 38 % 18 % N/A 

Selected Method==> BSFC Method 

Therefore, 18% would be selected as the best measurement allowance for PM, assuming 

it validates. Otherwise, the 38 % will be chosen if it validates. Thus, the additive brakespecific measurement allowance would be: 

PM = 18 % * 0.02 g/hp-hr = 0.0036 g/hp-hr, if it validates, and if not, then: 

PM = 38 % *0.02 g/hp-hr = 0.0076 g/hp-hr, if it validates, and if not, then: 

spend up to a $100,000 to figure out why it did not validate in the first place, and then 

apply the above strategy again, assuming the value now validates. If not, then EPA and 

EMA executive management will decide on the PM measurement allowance value. 

This PM value would be the value added to the actual brake-specific NTE threshold for a 

given engine, based on actual family emissions limit, mileage, model year, etc. 

2.3 Methods and Materials 

Exercise the model using three different calculation methods: a) Torque-Speed method, 

b) BSFC method, and c) ECM-Fuel Specific method (only for AVL MSS). Determine 

which calculation method is the most accurate, and use it to estimate the measurement 

allowance. Each calculation method is described in Figure 2, 3, and 4. 

Prepare an Excel spreadsheet model for use with the Crystal Ball Monte Carlo software 

for error analysis of brake specific emissions, BSE, as outlined in section 2.4. Changes to 

the model specifications may be requested as agreed upon by the Steering Committee. 

Prepare the spreadsheet in a modular structure following the specified model outline, and 

make provisions for the identified calculation modules. Additionally, clearly identify and 

easily locate input cells to the model to facilitate any revisions that may become 

necessary for users who want to exercise the model with other Monte Carlo add-ins such 

as @Risk or the newest versions of Crystal Ball. Test the spreadsheet with controlled test 

cases of simplified input distributions with the Crystal Ball add-in to confirm correct 

model implementation in accordance with this test plan. Run at least one typical analysis 

as an additional confirmation. 

Deliver the electronic spreadsheet and a brief report describing the model, presenting the 

test cases, and describing pertinent information including the Crystal Ball version number, 

the Excel version number, the operating system and the computer. Use standard 

15
 
 

 

spreadsheet calculations so that no serious difficulties will be anticipated regarding 

application in other spreadsheet versions. Use Crystal Ball Version 7 or higher, and 

confirm test cases using Excel 2003. 

Control revisions of the spreadsheet model using descriptive file names. Extensive 

revisions or testing with other software versions beyond that initially proposed may be reproposed by the Steering Committee if and when a need for such additional work is 

identified. 

2.4 Simulation Procedure 

For each of the measurement errors in Section 3, create an error surface and sample it 

according to the aforementioned PDFs. Each error surface represents an additive error— 

or a subtractive error if the sign is negative—relative to the reference value to which it is 

applied. Figure 5, Figure 6, and Figure 7 serve as a hypothetical PM example of how 

these error surfaces should be created for every error. The plots shown correspond to PM 

emissions concentration data representing 1 PEMS, two engines, and three exhaust 

configurations each, with all 6 sets of PEMS data pooled together. Note that separate 

error surfaces will be constructed for each of the three PEMS units (AVL, Horiba and 

Sensors). The example applies to the error module for steady-state (SS) bias and 

precision PM concentration errors (Section 3.2). These figures will be referenced by 

each “Data Analysis” section for the various errors discussed in this test plan. 

Errors from Section 3 (Engine Dynamometer Laboratory tests) and Section 4 

(Environmental Chamber Tests) are combined by adding all of the sampled errors once 

per NTE event trial. For example, in order to assess the errors in PM concentration for 

each NTE event, several modules will be created such that: 

PM_with errors = PM _ideal + μg/mole)1 + μg/mole)2 + μg/mole)3 + ... 

where, 

μg/mole)1 = PM concentration errors due to steady state bias and precision 

errors, 

μg/mole)2 = PM concentration errors due to ambient temperature, 

μg/mole)3 = PM concentration errors due to ambient pressure, 

etc.... 

2.4.1 Construction of the Error Surface 

2.4.1.1 PEMS vs. Lab 

Acquire raw data with the PEMS at various average concentration levels as per Section 

3.2. Plot the “PEMS” signals versus the corresponding “lab” signals that were measured 

using lab equipment. This plot pools all bias and precision errors for one PEMS and for 

all data from all engines for all steady-state modes. Shown in Figure 5 are the 5th, 50th 

16
 
and 95th percentiles at the mean PM concentration level from the lab (note that the 

distribution of data at each level is not necessarily Gaussian). If the 50th percentile is 

different than the line of perfect agreement (diagonal), the data suggests that there is a 

bias error between PEMS and Lab. In essence this graph shows the statistical distribution 

measured by the PEMS at each average concentration level sampled. The example shows 

only 6 discrete PM concentration levels (ranging from 10-60 μg/mole). However, the 

actual number of discrete levels will be determined by the total number of operating 

conditions actually run for all the tests of all the engines. For example, the SS PM testing 

will select 6 modes representing typical operating conditions. Thus, the actual plot for SS 

PM will likely have 36 discrete concentration levels (6 modes x 1 PEMS x 2 engines x 3 

exhaust configurations). 

0 

10 

20 

30 

40 

50 

60 

70 

80 

0 

PM μg/mole (PEMS) 

Error Surface for SS PM Concentration 

18 

26 

38 

45 

56 

69 

13 

24 

36 

42 

53 

64 

8 

19 

33 

39 

51 

61 

10 20 30 40 50 60 

PM μg/mole (lab,mean) 

70 80 

diagonal 95th percentile 

50th percentile (median) 5th percentile 

FIGURE 5. ERROR SURFACE: PEMS VS. LAB 

17
 
9 

Error Surface for SS PM Concentration 

PM μg/mole (PEMS) - PM μg/mole (lab) 

10 

8 

6 

4 

2 

0 

-2 

-4 

8 

6 

8 

5 

6 

3 

4 

6 

2 

3 

4 

-2 

-1 

3 

-1 

1 1 

0 10 20 30 40 50 60 70 

PM μg/mole (lab,mean) 

95th percentile 50th percentile (median) 5th percentile 

FIGURE 6. ERROR SURFACE: (PEMS-LAB) VS. LAB 

18
 
 

Error Surface for SS PM Concentration 

10.1 6.8 8.8 7.2 11.1 

3 4 6 2 3 4 

-4.1 -3.1 1.8 -2.2 0.2 -0.2 

-1.5 

-1 

-0.5 

0 

0.5 

1 

1.5 

0 10 20 30 40 50 60 70 

PM μg/mole (lab,mean) 

ic_SS_PM 

99th percentile 50th percentile (median) 1st percentile 

Error Surface: z-axis = ΔSS_PM_μg/mole 

6. 

FIGURE 7. ERROR SURFACE: FINAL VERSION 

19
 
2.4.1.2 (PEMS – Lab) vs. Lab 

The plot in Figure 6 basically shows the “additive error band” measured during testing. 

The plot is created by first subtracting the “lab” PM value from the corresponding 

individual PEMS PM measurement for each test run. This difference is defined as the 

‘delta’ error. Next, the “PEMS – Laboratory” delta errors are pooled at each average lab 

PM value to obtain the 95th, 50th, and 5th percentile values, respectively, displayed in 

Figure 5. Notice that if lab error exceeds PEMS error at a given percentile, crossover of 

values can occur. This is acceptable because the crossover effectively reduces PEMS 

error whenever lab error exceeds PEMS error. 

In order to obtain estimates of the 1st and 99th percentiles for the delta errors for a given 

“lab” PM value, each side of the corresponding error distribution will be assumed to 

independently fit a normal distribution. Because of the asymmetry of the data, this 

methodology will yield two halves of a normal distribution. The median of each normal 

distribution will be the median based on the delta errors given in Figure 6. The 95th 

percentile delta error will form the upper boundary of one half of the normal distribution, 

and the 5th percentile delta error will form the lower boundary of the other half of the 

normal distribution. When each side of the data distribution is fitted to a normal 

distribution using the above boundary conditions, one can then expand each half of the 

distribution from the error surface to obtain the 1st and 99th percentiles of the data for the 

given “lab” PM value. 

2.4.1.3 Error Surface 

This step normalizes the data in Figure 7 using what is called a “variability index (ic)”, 

which represents the random sampling by the Monte Carlo technique, in order to select a 

given error level. This variability index is allowed to vary from –1 to +1. The likelihood 

of ic being any value between –1 through +1 is specified by the PDF assigned to ic. In the 

given example, ic is assumed to vary according to a normal distribution during Monte 

Carlo calculations. This is because it is believed that the distribution of errors due to 

steady-state bias and precision will be centered about the 50th percentile of the full range 

of conditions measured according to Section 3.2. The pressure and temperature 

environmental error modules use uniform probability density functions for their 

respective variability index. Each set of data for each lab set-point mean (i.e., lab 

reference value) in Figure 6 is normalized by aligning the 1st percentile error from the 

fitted normal distributions with ic = -1, the 50th percentile error with ic = 0, and the 99th 

percentile error from the fitted normal distribution with ic = +1. 

Error surfaces such as the one presented in Figure 7 are the input modules that the Monte 

Carlo simulation program will use during calculations of brake-specific PM emissions. 

For example, for a given NTE calculation a random ic value is chosen once per NTE 

event trial. Let us assume that the first random sample produced an ic = 0.5. Let us also 

20
 
 

 

assume that during this NTE event trial, the reference PM concentration is 10 μg/mole. 

In this case, 

μg/mole)1 = (3 + 10.1) / 2 = 6.55 μg/mole. 

Also, from Figure 7, for ic = 0.5, the reference PM = 10 μg/mole. 

For that step in the calculation, the Monte Carlo approach will add this “delta” to the 

reference concentration value of 10 μg/mole (10 μg/mole + 6.55 μg/mole = 16.55 

μg/mole) to represent errors in steady-state bias and precision for ic = 0.5, and reference 

NTE PM = 10 μg/mole. If during the same NTE event in the reference data set, a 

reference concentration of 35 μg/mole is read, then, 

μg/mole)1 = ((6 + 8.8) / 2 + (2 + 6.2) / 2) / 2 = 5.75 μg/mole (from 

Figure 7) 

Note that first the error along the ic line perpendicular to the ic axis (in this case the line 

along 0.5) is linearly interpolated at each discrete concentration level. Then those 

interpolated values are themselves linearly interpolated to determine the error 

corresponding to each reference concentration in the NTE event. Note that the random 

selection is once per reference NTE event trial, but the error along that ic line is applied to 

every second-by-second value within the given reference NTE event, except for PM 

concentration in the case of Horiba and Sensors, where no second-by-second information 

are available, but different PM concentration levels may be available for a specific NTE 

event. 

Now let us assume that the error in PM concentration is composed of only 3 deltas: 

μg/mole)1, μg/mole)2 , and μg/mole)3 . And let us assume that for a given 

reference NTE event trial we have the following values: 

 Reference PM at one second= 30 μg/mole 

 μg/mole)1 = 6 μg/mole 

 μg/mole)2 = -2 μg/mole 

 μg/mole)3 = -3 μg/mole. 

When the model calculates brake-specific emissions by each of the three calculation 

methods, it will use the following PM value, which has all of its error applied: 

PM = 30 + 6 –2 – 3 = 31 μg/mole. 

The application of error at the first selected ic continues during the entire NTE event 

without having to randomly sample again. In other words, ic will not change during that 

random trial. For all of the variables except for mPM , the errors may continue to change 

during an NTE event on a second-by-second basis if their error surface happens to be a 

function of level. For the second randomly selected ic this entire process of determining 

21
 
the μg/mole errors is repeated. The simulation will continue to randomly selected ic 

values for thousands of trials until convergence is met. 

For the Horiba and Sensors generated reference NTE events, there is only one flowweighted PM value for the entire NTE event. During the simulation for these types of 

reference NTEs, the single PM value will be used in the interpolation of the 

corresponding PM error surfaces (i.e., steady-state PM, transient PM) at all seconds of 

the reference NTE event. Since the PM value will not vary from second-to-second, the 

only interpolation will occur according to the ic value at each of the simulation trials. 

The same second-by-second sampling and interpolation approach would be used for other 

deltas such as ambient temp, ambient pressure, shock and vibration, BSFC interpolation, 

torque, exhaust flow rate, etc. An overview of the Monte Carlo simulation for PM is 

detailed in Figure 8. 

FIGURE 8. OVERVIEW OF MONTE CARLO SIMULATION 

Table 3 lists the error surfaces that will be created for use in simulating the BSPM error 

differences. 

22
 
TABLE 3. ERROR SURFACES FOR THE BSPM SIMULATION
 

Calculation 

Component Test Source Error Surface 

Delta PM Engine Dyno Delta PM SS 

Delta PM Engine Dyno Delta PM Transient 

Delta PM Environ Delta PM Ambient Temperature 

Delta PM Environ Delta PM EMI/RFI 

Delta PM Environ Delta PM Atmospheric Pressure 

Delta PM Environ Delta PM Vibration 

Delta CO Engine Dyno Delta CO SS 

Delta CO Environ Delta CO Atmospheric Pressure 

Delta CO Environ Delta CO Ambient Temperature 

Delta CO Engine Dyno Delta CO Time Alignment 

Delta CO2 Engine Dyno Delta CO2 SS 

Delta CO2 Engine Dyno Delta CO2 Transient 

Delta CO2 Environ Delta CO2 Ambient Temperature 

Delta NMHC Engine Dyno Delta NMHC SS 

Delta NMHC Engine Dyno Delta NMHC Transient 

Delta NMHC Environ Delta NMHC Atmospheric Pressure 

Delta NMHC Environ Delta NMHC Ambient Temperature 

Delta NMHC Environ Delta Ambient NMHC 

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow SS 

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow Transient 

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow Pulsation 

Delta Exhaust Flow Engine Dyno Delta Exhaust Flow Swirl 

Delta Exhaust Flow Environ Delta Exhaust EMI/RFI 

Delta Exhaust Flow Environ Delta Exhaust Temperature 

Delta Exhaust Flow Environ Delta Exhaust Pressure 

Delta Torque Engine Dyno Delta Dynamic Torque 

Delta Torque Engine Dyno Delta Torque DOE Testing 

Delta Torque Engine Dyno Delta Torque Warm-up 

Delta Torque Engine Dyno Delta Torque Humidity/Fuel 

Delta Torque Engine Dyno Delta Torque Interpolation 

Delta Torque Engine Manuf Delta Torque Engine Manuf 

Delta Speed Engine Dyno Delta Dynamic Speed 

Delta Fuel Rate Engine Dyno Delta Dynamic Fuel Rate 

23
 
2.5 Model Considerations 

2.5.1 Convergence 

The main goal of the convergence criteria is to define how many simulation trials at a 

given reference NTE event are required to estimate the 95th percentile BSPM emission 

differences with a given precision. The convergence method to be used is based on a 

nonparametric statistical technique3

 which defines a 90% confidence interval for the 95th 

percentile of the BSPM emissions differences for an individual reference NTE 

simulation. If the width of the 90% confidence interval is less than 1% of the BSPM 

emissions threshold, then convergence is met. The following steps define the 

convergence method: 

1.	 Run the Monte Carlo simulation for N trials for a single reference NTE event. 

2.	 Order the BSPM emissions differences from smallest to largest. 

3.	 Identify the trial number at the lower end of the 90% confidence interval 

nlower = 0.95 * N 1.645 0.95 * 0.05 * N

4.	 Identify the trial number at the upper end of the 90% confidence interval 

nupper = 0.95 * N 1.645 0.95 * 0.05 * N

5.	 Compute (BSPM difference value at nupper) – (BSPM difference value at nlower). 

6.	 If the result in (5) < 1% of the BSPM emissions NTE threshold (0.02 g/hp-hr) then 

convergence is met. 

2.6 Simulation Output 

It is important to understand and identify what error surfaces have the most influence 

(i.e., sensitivity) on the BSPM emissions ‘with errors’ and, thus, the resulting BS 

emissions differences. Contributions to sensitivity can be attributable to changes in 

variance and/or bias. 

2.6.1 Sensitivity Variation Effect 

During the Monte Carlo simulation for each reference NTE event, sensitivity charts 

produced by Crystal Ball will be generated and stored in output REPORT files. Crystal 

Ball calculates sensitivity by computing the rank correlation coefficient between every 

assumption (error surface) and forecast value (delta BS emissions) while the simulation is 

running. Positive rank correlations indicate that an increase in the assumption is 

associated with an increase in the forecast. The larger the absolute value of the rank 

correlation the stronger the relationship. 

Sensitivity charts in Crystal Ball provide a means to determine how the variances of the 

error surfaces affect the variance in the forecast values. Hence, the sensitivity charts 

developed during a simulation are displayed as “Contribution to Variance” charts which 

24
 
are calculated by squaring the rank correlation coefficients for all assumptions used in a 

particular forecast and then normalizing them to 100%. The assumption (error surface) 

with the highest contribution to variance (in absolute value of the percent) is listed first in 

the sensitivity chart. 

Simulation results from all reference NTE events will produce sensitivity values for the 

95th percentile delta PM emissions by all three calculation methods. 

2.6.2 Sensitivity Bias Effect 

Another type of sensitivity to be examined in this study is concerned with the effects of 

potential “bias” in error surfaces and their effects on the forecast values. In order to 

study these effects a new error surface assumption will be added to the simulation model 

for each of the original error surfaces. 

This assumption will be sampled as a discrete binary distribution (i.e., on or off) during 

the simulation. For each trial of the simulation, the original error surfaces and ‘on/off’ 

error surfaces will be sampled according to their defined sample distribution. If the 

‘on/off’ error surface produces an ‘off’ condition, the delta emissions from that particular 

error surface will not be added to the BSPM emissions computations for the BSPM 

emissions ‘with errors’. Similarly, if the ‘on/off’ error surface produces an ‘on’ 

condition, the delta emissions from that particular error surface will be added to the 

BSPM emissions calculations. 

During every trial of the simulation, the exclusions due to the ‘off’ conditions will result 

in various combinations of the error surface delta emissions being added to the BSPM 

emissions ‘with errors’ computations. Over the course of a simulation with thousands of 

trials, the sensitivity of a particular error either ‘on’ or ‘off’ will be assessed by 

examining the change in the forecast delta emission. Therefore, in a single Monte Carlo 

simulation of a reference NTE event sensitivities due to variance and/or bias will be 

explored. 

25
 
3 ENGINE DYNAMOMETER LABORATORY TESTS 

Utilize engine dynamometer laboratory testing to establish the difference between PM 

PEMS and PM based on laboratory measurement in accordance with Part 1065. Also 

establish how well ECM parameters can be used to estimate torque and BSFC. 

First, however, audit all the PEMS and lab equipment to ensure that they are operating 

properly, according to 40 CFR Part 1065, Subpart D. Next, conduct steady-state engine 

dynamometer tests to establish PEMS steady-state bias and precision relative to the lab. 

Then, conduct transient engine dynamometer testing to determine PEMS transient 

precision by repeating transient NTE events. Finally, compare ECM derived torque and 

BSFC to laboratory measured torque and BSFC. 

3.1 Preliminary Audits 

3.1.1 Objective 

Conduct 40 CFR Part 1065, Subpart D audits of all engine dynamometer laboratory 

systems and all PEMS. 

3.1.2 Background 

Because the overall purpose of this entire test plan is to establish measurement allowance 

that account for the incremental difference in the performance of PEMS versus engine 

dynamometer laboratory systems, the first task is to audit all of the measurement systems 

to ensure that the specific systems used for testing meet EPA’s minimum performance 

requirements. The audits also help to minimize bias errors between PEMS and lab 

systems measurements. However, in case a specific PM-PEMS does not meet the 

specifics of Part 1065 requirement, the MASC will decide on how to move forward by 

perhaps allowing some flexibility in passing Part 1065 audit, in situations where it might 

be needed, especially if the performance of a system is within the expectation of the 

manufacturer. 

3.1.3 On-site meeting to establish 1065 compliance requirements 

In order to clarify what are all the requirements expected from the lab-grade 

instrumentation and PEMS equipment, with respect to 1065 compliance, a meeting will 

be held between the test plan steering committee and the contractor at the contractor site 

to provide the contractor with guidance regarding which specific sections of Part 1065 

Subpart D are required and which are optional. In case Part 1065 requirement is 

demonstrated to be too stringent or impractical, the contractor may seek approval from 

the MASC to lessen the stringency of Part 1065 in relation to the PEMS. 

3.1.4 Methods and Materials 

Use the methods and materials described in 40 CFR Part 1065, Subpart D to conduct 

audits of all lab and PEMS measurement systems. Even if lab systems and PEMS pass 

initial Subpart D audits, allow lab operators and PEMS manufacturers to make on-site 

26
 
adjustments to improve the performance of their systems prior to engine testing. Allow 

adjustments to be based on recalibrations with reference signals that are allowed in 40 

CFR Part 1065. The steering committee may direct the contractor to calibrate or adjust 

the laboratory sampling system based on audit results. The steering committee may also 

suggest that a PEMS manufacturer calibrate or adjust one or more PEMS based on lab 

audits. 

3.1.5 Data Analysis 

Use the data analyses described in CFR Part 1065 Subparts D, J and G. For all 

subsequent testing, use only those measurement systems that pass the minimum 

performance criteria in Subpart D, unless a deficiency is deemed acceptable in writing by 

all parties including PEMS manufacturers. Provide a list and brief description of all the 

audits conducted for each PEMS manufacturer type. EPA would likely use this list as a 

template for the data requirements in the PM portion of the HDIU testing program. 

3.1.6 PEMS Manufacturer PM PEMS Commissioning 

Notify PEMS manufacturers when the 1065 audits are complete and the first set of PM 

PEMS are completely installed in the engine dynamometer test cell—in preparation for 

emissions testing. Schedule dates and times that are prior to the start of emissions testing 

for each PEMS manufacturer to conduct a final commissioning of all their PEMS that are 

on site, including those PEMS that are not installed in the test cell. PEMS manufacturers 

may inspect their PEMS and make any final adjustments to their respective PEMS in 

order for the PEMS to meet their specifications. Allow PEMS manufacturers to inspect 

the installation of their PEMS in the test cell. If PEMS manufacturers take exception to 

any portion of the installation or on-site configuration, attempt to resolve any such 

installation issues. If such issues are not easily resolvable, notify the steering committee, 

who will determine a course of action. Once PEMS manufacturers have completed their 

commissioning, notify the steering committee. From this point any further modifications 

to the PEMS may only be made according to Table 1 of this test plan. 

3.2 Bias and Precision Errors under steady state engine operation 

3.2.1 Objective 

Evaluate the bias and precision using one engine and one exhaust configuration, shown in 

Table 4, and 10 repeats of steady-state modes, and three sets of PEMS units, each set 

including the MSS, TRPM, and PPMD. Thus, the total number of NTE steady-state 

points required to conduct the steady-state experiments is 30. This constitutes six steadystate modes of engine operation (6), 10 repeats (10), one exhaust configuration, one 

engine (1), and three different PEMS units (3), 6x10x1x1x3= 180. 

g Determine the ΔSSmPM  mol surface plots for the error model based upon all data pooled. 

Note that each brand of PEMS will have its own ΔSSmPM error surface generated for use 

27
 
in both calculation methods 1 and 2. For calculation method 3, the AVL brand PM 

PEMS will have a unique Δ m calculated according to Figure 4 of this test plan. SS PM 

 Recommend six steady-state points based on the PM measurement, using the AVL MSS, 

of 80 SS points of the Cummins cycle that is typically used to generate ECM torque and 

BSFC errors versus laboratory. The MASC will accept the six steady-state points or 

choose alternative points for each exhaust configuration. The objective for the MASC 

will be to select steady-state points within a given exhaust configuration that provides a 

nominal spread of concentrations within that configuration’s target brake-specific levels. 

Note that to achieve the brake-specific targets under steady-state conditions, the bypass 

might have to be opened further, relative to the transient NTE bypass settings. 

TABLE 4. ENGINE, EXHAUST CONFIGURATION, AND STEADY-STATE
 

MODES 

No. of Steady-State 

Modes for Bypass 

Setting 1 (BSPM and 

PM Concentration, 

representative of PM 

threshold of 0.025 

g/hp-hr under NTE 

Transient Operation) 

PM-PEMS Units Number of Repeats 

07 Engine 1 SS1, SS2, SS3, SS4, 

SS5, SS6 

Three Sets of (MSS, 

TRPM, and PPMD) 

10 per Mode per PMPEMS Set 

3.2.2 Background 

Testing will be conducted to capture bias and precision errors in PEMS’ emissions 

instruments versus the laboratory filter-based method. The tests will be steady-state only. 

Note: Section 3.3 (next section) will evaluate precision errors (not bias) due to the 

dynamic response of the PEMS instrumentation. The precision error captured during 

steady state testing (section 3.2) will have to be subtracted from the overall precision 

error captured in section 3.3 in order not to double-count the steady state precision errors 

of PEMS instrumentation. This process is detailed in Section 3.3. 

3.2.3 Methods and Materials 

Use the following systems:
 

a) One model year 2007 heavy duty diesel engines, equipped with a DPF in the exhaust 

(Mack MP9) 

b) Nine PM PEMS (3 Sensors PPMD, 3 AVL MSS, 3 Horiba TRPM) 

c) One PEMS exhaust flow-meter from Sensors, Inc., and one and from Horiba, 

applicable to the engine to be tested 

28
 
d) DPF with Bypass Setting 1 for SS testing, representing a threshold level of about 

0.025 g/hp-hr under NTE transient testing 

Use the following overall guidelines: 

e)	 Measure PM via the CVS, Part 1065 Lab Method (most recent publication) 

f)	 Measure engine inlet airflow through use of LFE or equivalent 

g)	 Use a series of six steady-state modes, and set each mode time to collect a CVS filter 

mass of at least 75 microgram per mode, simultaneously with other PM-PEMS 

h) Regenerate DPF system prior to each series of steady-state tests 

i)	 Capture ECM broadcast channels and other common diagnostic channels, as 

recommended by engine manufacturer(s), to ensure proper engine operation 

j)	 Do not measure gaseous species by the PEMS 

k)	 Stabilization time = 180 seconds, with a different running time per mode to achieve a 

75 microgram or higher of PM on the CVS filter 

l)	 Always power off PEMS equipment at end of each day, according to PEMS 

manufacturer instructions. Re-start start-up process every day according to PEMS 

manufacturer instructions and Part 1065, Subpart J. 

m) Whenever PEMS are exchanged, swap the order of the Horiba and Sensors 

flowmeters, if the steup allows for it. 

6 point steady-state repeat-testing, evaluate bias and precision errors: 

a)	 The MASC will select 6 SS operating conditions for repeat testing from a matrix of 

80 SS points containing information on PM emissions using the AVL MSS 

b)	 Randomize the order of the six modes 

c)	 Repeat each six steady-state cycle two or three times, prior to DPF regeneration 

d)	 Each test will use three PEMS (Sensors, AVL, and Horiba) at a time, to measure PM 

emissions concentration and exhaust flow rate. 

e)	 Expected test duration is 5 days per PEMS set, with a total of 15 days for all three 

sets. 

Bypass Setting: 

a)	 Run NTE transient cycle using the CVS filter-based method 

b)	 Set bypass to produce CVS filter-based average brake-specific of about 0.025 g/hp-hr 

c)	 Determine the average PM mass concentration 

d)	 Run the 80 SS Cummins cycle to capture PM concentration at each mode using the 

AVL MSS 

e)	 Check the PM concentration levels and select the six-steady state modes from the 80 

point matrix. As a first order, check the concentration at the pre-selected steady-state 

modes to see if they spread within reason around the concentration produced for the 

NTE transient cycle. If not, adjust the bypass as needed to establish the right spread in 

brake-specific emissions and concentration for the six steady-state modes 

29
 
 

 

 

 

 

 

f)	 Make sure that the points selected spread around a brake specific level and 

concentration level of a threshold of 0.025 g/hp-hr, and concentration range of 4 to 15 

milligram per cubic meter. 

3.2.4 Data Analysis 

Use the acquired data to create the “error surfaces” to be used by the Monte Carlo 

simulation. An example of the steady-state error surface determination is shown in Table 

5 for PM. 

TABLE 5. EXAMPLE OF SS ERROR SURFACE 

Error Surface for SS PM Concentration 

Figure 5 

x-axis PM μg/mole (lab mean at setpoint) 

y-axis PM μg/mole (PEMS) 

Figure 6 

x-axis PM μg/mole (lab mean at setpoint) 

y-axis 5th percentile 5th [PM μg/mole (PEMS) - PM μg/mole (lab)] 

y-axis 50th percentile 50th [PM μg/mole (PEMS) - PM μg/mole (lab)] 

y-axis 95th percentile 95th [PM μg/mole (PEMS) - PM μg/mole (lab)] 

The 5th, 50th and 95th percentiles from the (PEMS - lab) delta data will be used to 

estimate the 1st and 99th percentiles from assumed Gaussian distributions. 

Figure 7 

x-axis PM μg/mole (lab mean at setpoint) 

y-axis ic_SS_PM 

z-axis = ΔSS_PM_μg/mole 1st Percentile from Gaussian distribution based on 5th 

and 50th [PM μg/mole (PEMS) - PM μg/mole (lab)] 

deltas. 

99th Percentile from Gaussian distribution based on 50th 

and 95th [PM μg/mole (PEMS) - PM μg/mole (lab)] 

deltas. 

50th Percentile based on [PM μg/mole (PEMS) - PM 

μg/mole (lab)] deltas. 

ic sample frequency once per NTE event trial 

ic sample distribution Gaussian (normal distribution) 

30
 
3.3 Precision Errors under transient engine operation (dynamic response) 

3.3.1 Objective 

The objective of this portion of the work is to determine the precision error, Δ m  TR PM 

with each PM-PEMS under NTE transient engine operation. This will be achieved by 

creating a 20 to 25-minute transient NTE cycle where the PEMS measure in each NTE. 

3.3.2 Background 

PEMS are expected to operate in a repeatable manner over NTE events as short as 30 

seconds. Two sources of PEMS precision error are hypothesized: 1) dynamic response to 

rapidly changing signals, and 2) susceptibility to “history” effects. Dynamic response 

error includes error due to measurement signal time alignment, and the dissimilarity of 

the dynamic response and aliasing of signals; including those signals used to determine 

entry into and exit from the NTE zone. History effects include the effects of previously 

measured quantities on currently measured quantities. For example, this may be caused 

by ineffective sample exchange in the PM emissions sampling volumes, or it may be 

caused by one or more sensors’ characteristic rise time or fall time. To account for any 

dynamic response precision error, the increase in precision error incremental to the 

steady-state emissions measurement precision will be incorporated into the overall error 

model. 

Selection of short NTE cycles (each 32 seconds) maximizes the sensitivity of this test to 

effects of dynamic response. Thirty-two seconds was chosen as the minimum instead of 

thirty seconds, which is the shortest NTE event time, to ensure that 1 Hz ECM updating 

of torque and speed values would be unlikely to interfere with capturing NTE events. For 

each repeat of the test cycle, the order of the 30 different NTE events will be the same. 

In addition the 29 different intervals separating each NTE event from the next will have a 

range of durations and these will be randomly arranged in each test cycle as well. Fixed 

arrangement of the NTE events and the inter-NTE events will maximize the sensitivity of 

this test to dynamic response and history effects, and make the DPF and bypass operation 

very consistent. 

The total length of the NTE transient cycle will assume that only 5 quartz crystal of the 

Sensors PPMD are working, and it takes five minutes of stabilization time for reusing a 

crystal after PM collection. Thus, the same NTE transient cycle used in the gaseous 

PEMS program will be used here, except for changes in the inter-NTE times to 

accommodate the Sensors PPMD. 

3.3.3 Methods and Materials 

a. Use a transient engine dynamometer emissions laboratory. 

31
 
 

b.	 Use a laboratory that can accommodate at least three PEMS, their power supplies, 

the PEMS flow meters, cables and lines. 

c.	 Use same overall guidelines described in section 3.2, but applied to transient 

engine testing. 

d.	 Record the EEPS’ total mass signal during transient testing. 

Challenge PEMS to 30 different 32-second NTE events, shown in Table 5, over about 23 

minute test cycle, or whatever needed to accommodate the need for five crystals of the 

PPMD to be operational. Randomize the NTE events shown in Table 6 once, scale up 

every fifth inter-NTE time, shown in Table 7, to accommodate the PPMD, and use the 

same order for repeat testing. Repeat the test cycle 10 times for each set of three PEMS. 

Note that for any torque command that is less than zero, command closed throttle (i.e. 

zero or minimum fuel command), and motor the engine at the commanded speed for that 

data point. An example of an NTE transient cycle is shown in Figure 9. 

Based on 10 repeats with each set of PEMS, the total number of repeats will be 30 cycles, 

assuming 1 NTE cycle x 10 repeats x one exhaust configuration x 3 sets of PEMS x one 

engine (1x10x1x3x1 = 30). Assuming a 25 minutes of NTE with 30 minutes of forced 

regeneration and preparation for the second repeat, the total number of days for NTE 

transient testing is 10 days (8 hours per day). This time includes PEMS and engine setup, 

PEMS warm up, and daily checks. 

TABLE 6. NTE TRANSIENT CYCLE 

NTE 

Event 

1

Speed % 

Range 

2

Torque % 

Range 

Description 

NTE1 17% 3

32% Steady speed and torque; lower left of 

NTE 

NTE2 59% 3

32% Steady speed and torque; lower center of 

NTE 

NTE3 Governor line 3

32% Steady speed and torque; lower right of 

NTE 

NTE4 17% 66% Steady speed and torque; middle left of 

NTE 

NTE5 59% 66% Steady speed and torque; middle center of 

NTE 

NTE6 Governor line 66% Steady speed and torque; middle right of 

NTE 

NTE7 17% 100% Steady speed and torque; upper left of 

NTE 

NTE8 59% 100% Steady speed and torque; upper center of 

NTE 

NTE9 100% 100% Steady speed and torque; upper right of 

NTE 

NTE10 Lower third 3

32% - 100% Highly transient torque; moderate transient 

32
 
speed 

NTE11 Upper third 3

32% - 100% Highly transient torque; moderate transient 

speed 

NTE12 Middle third 3

32% - 100% Highly transient torque; moderate transient 

speed 

NTE13 17% - governed Lower third Highly transient speed; moderate transient 

torque 

NTE14 17% - governed Upper third Highly transient speed; moderate transient 

torque 

NTE15 17% - governed Middle third Highly transient speed; moderate transient 

torque 

NTE16 Lower right diagonal Transient; speed increases as torque 

increases 

NTE17 Upper left diagonal Transient; speed increases as torque 

increases 

NTE18 Full diagonal; lower left to upper 

right 

Transient; speed increases as torque 

increases 

NTE19 Lower left diagonal Transient; speed decreases as torque 

increases 

NTE20 Upper right diagonal Transient; speed decreases as torque 

increases 

NTE21 Full diagonal; lower right to upper 

left 

Transient; speed decreases as torque 

increases 

NTE22 Third light—heavy-duty NTE 

event from International, Inc. data 

set 

Sample from LHDE 

NTE23 Cruise; ~ 50 mph Sample from HDDE 

NTE24 Cruise; ~ 75 mph Sample from HDDE 

NTE25 Small bulldozer Sample from NRDE 

NTE26 Large bulldozer Sample from NRDE 

NTE27 Second of three NTE events in 

FTP 

Seconds used from FTP: 714-725, 729­

743, 751-755 

NTE28 Third light—heavy-duty NTE 

event from International, Inc. data 

set 

Sample from LHDE 

NTE29 First of two NTE events in NRTC Seconds used from NRTC: 423-430, 444, 

448-450, 462-481, increased 464 speed 

from 40% to 42% 

NTE30 First of two NTE events in NRTC Seconds used from NRTC: 627-629, 657­

664, 685-696, 714-722 

1 Speed (rpm) = Curb Idle + (Speed % * (MTS - Curb Idle) 

2 Torque (lbf-ft) = Torque % * Maximum Torque At Speed (i.e. lug curve torque at 

speed) 

3 Torque (lbf-ft) = Maximum of (32 % * peak torque) and the torque at speed that 

produces (32 % * peak power) 

33
 
TABLE 7. DYNAMIC RESPONSE INTER-NTE EVENTS 

INT 

Event1 

Duration 

(s) 

Frequency Description 

INT1 10 1 Initiation of cycle; INT1 is always first 

INT2-6 2 5 Shortest and most frequent inter-NTE events 

INT7-10 3 4 Short and frequent inter-NTE events 

INT11-14 4 4 Short and frequent inter-NTE events 

INT15-18 5 4 Short and frequent inter-NTE events 

INT19-21 6 3 Short and frequent inter-NTE events 

INT22 7 1 Medium inter-NTE event 

INT23 8 1 Medium inter-NTE event 

INT24 9 1 Medium inter-NTE event 

INT25 11 1 Medium inter-NTE event 

INT26 13 1 Long inter-NTE event 

INT27 17 1 Long inter-NTE event 

INT28 22 1 Long inter-NTE event 

INT29 27 1 Long inter-NTE event 

INT30 35 1 Longest inter-NTE event 

INT31 5 1 Termination of cycle; INT31* is always last 

Interval speeds and torques are not identical, but they are clustered around zero torque 

and the speed at which 15% of peak power and 15% of peak torque are output. 

34
 
Torque (lbf-ft) 

Torque-Speed Domain 

700 

Lug Curve 

NTE S line 

600 NTE T-S line 

1 SS Lower Left 

2 SS Lower Mid 

3 SS Lower Right 500 

4 SS Mid Left 

5 SS Mid Mid 

6 SS Mid Right 400 7 SS Upper Left 

8 SS Upper Mid 

9 SS Upper Right 

300 NTE Cycle 

22 LD NTE3 

23 OH 50mph Cruise 

200 24 OH 70mph Cruise 

25 NR Small Dozer 

26 NR Large Dozer 

27 FTP NTE2 100 28 LD NTE6 

29 NRTC NTE1 

30 NRTC NTE2

0 

0 500 1000 1500 2000 2500 3000 3500 4000 4500 

-100 

Speed (rpm) 

FIGURE 9. EXAMPLE OF A NTE CYCLE 

Prior to executing the first repeat, setup each PEMS and stabilize engine operation at the 

first inter-NTE operating point. Setup the PEMS according to 40 CFR Part 1065 and 

PEMS manufacturer instructions, including any warm-up time, zero-spans of the 

analyzers and the setup of all accessories including flow meters, ECM interpreters, etc. 

Then, when the test cycle starts, switch the PEMS’ to sample emissions from the engine. 

When the text cycle ends, switch the PEMS back to ambient sampling. Complete all 

post-test lab and PEMS validations according to 40 CFR Part 1065 and according to 

PEMS manufacturer instructions. 

3.3.4 Data Analysis 

Discard from further data analysis any NTE events invalidated by any criteria in 40 CFR 

Part 1065 Subpart J. For each NTEi event (i=1 to 30), which was repeated 30 times per 

engine with a specific exhaust configuration (j = 1 to 30), calculate the transient median 

absolute deviation, MADTRi, for mPM , where for each NTEi event, MADTRi = median[|

NTEij – median (NTEij) |]. 

Next calculate the difference of MAD by subtracting a corresponding steady-state MAD,

MADSSi for mPM . MADTRi-SSi = MADTRi – MADSSi. To determine a corresponding 

MADSSi, calculate the PEMS MADSS at each steady-state median lab value, and then use 

the median PEMS NTEi value along the median lab value’s axis to find MADSSi for the 

corresponding MADTRi . Do not extrapolate any MADSSi beyond the minimum or

35
 
maximum median lab values. Note that some MADSSi values might be zero because the 

lab data for that median failed the F-test in the previous section. 

For any MADTRi-SSi less than zero, set that MADTRi-SSi equal to zero. 

Create a transient error surfaces using all of the MADTRi-SSi. Be sure to include any 

MADTRi-SSi data points that are equal to zero because they will affect the 1st and 99th
 

percentile values. 

36
 
3.4 ECM Torque and BSFC 

3.4.1 Objective 

Compare the ECM-based torque and fuel rate with that of the laboratory-based 

measurement using the Cummins 80 SS mode cycles. For the laboratory purposes, use 

the gas-based fuel flow values instead of the measured fuel flow. Repeat the Cummins 80 

SS cycle three times, and use the average values produced. 

Use at least six engines for these experiments that include the one engine to be used in 

the PM PEMS program and Engine B, C, and D of the ACES program. 

3.4.2 Data Analysis 

Use the acquired data pooled and normalized to % of max torque and % of maximum fuel 

rate to replace the manufacturer submitted error surfaces that were previously used in the 

gaseous portion of the Monte Carlo simulation. Refer to section 2.4 for description and 

example of an error surface. Include any bias error, unless there is an assignable cause 

that would not occur in-use and the steering committee approves to eliminate such bias 

error. 

4 ENVIRONMENTAL CHAMBER 

The environmental chamber tests challenge PEMS to a variety of environmental 

disturbances, namely electromagnetic interference, atmospheric pressure, ambient 

temperature and humidity, and shock and vibration. 

During each of the tests, plus a baseline test, the PEMS will cycle through sampling four 

different dilution preparations of aerosol particles that contain volatile hydrocarbon and 

elemental carbon using a particle generator that mimics the formation of diesel particles. 

The OC/EC will be used to determine the concentration levels needed for the PM 

generator. Essentially, after determining the steady-state points to run on the engine, the 

OC/EC semi-continuous instrument will be used along with the filter-based method. Then, 

for the concentration levels to be used with the PM generator, the OC/EC instrument will 

be used to set the PM generator to produce the desired composition and concentration 

levels, similar to those encountered under steady-state. Three particle concentration 

levels of 5, 10, and 15 mg/m3

, as shown in Table 8, will be generated by the particle 

generator. Each concentration will be fed to the PEMS after applying dilution ratios of 6, 

12, 20, and 30. For each concentration and dilution ratio combination, the PM generator 

will be stabilized for 4.5 minutes, and data will be collected by the PEMS for 30 seconds. 

The test will continue for a period of 8 hours. The first six cycles of every test will serve 

to be the baseline before any environmental change is made. 

37
 
   

   

   

         

 

The temperature/humidity and pressure tests are designed to mimic real-world 

environmental disturbances with the magnitude and frequency of the disturbances 

adjusted to real-world conditions. Randomly sample a uniform distribution of probability 

for their ic. , from any minute of the test. By randomly sampling from the minutes of 

these tests the magnitude and frequency of the real-world error will be built into the error 

model, which is described in Section 2. The other environmental tests represent the full 

range of possible conditions. For these tests, randomly sample the normal distribution in 

Figure 1 for their ic. 

For EMI/RFI and vibration, the instruments will be subjected to screening tests with 

HEPA filtered air to detect if there any changes in the response of the instruments. Based 

on these results, the MASC will decide if the particle generator will need to be used with 

these tests. 

For the vibration screening test, in order to avoid damage to the instruments, a frequency 

sweep will be used at low amplitude. The idea here is to detect the frequency that may 

trigger a response by the instrument, without doing any damage due to high amplitude. 

 TABLE 8. CONCENTRATION AND DILUTION RATIO SCHEDULE WITH PM 

GENERATOR 

Raw PM Concentration,

Dilution Ratio

DR1 DR2 DR3 DR4

6 12 20 30

g/m3 Concentration at Above Dilution Ratio

 833.3 416.7 250.0 166.7

10000 1666.7 833.3 500.0 333.3

15000 2500.0 1250.0 750.0 500.0

4.1 Data Analysis for Environmental Tests 

Reduce data by first calculating means for each 30-second period of stabilized 

measurements. Subtract from each mean the respective baseline concentration. The 

results are errors or “deltas”. Correct each of these error distributions by removing their 

respective baseline variances, which were determined by quantifying PM Generator 

output with no environmental perturbations. Calculate the variance of each of the 

distributions. Subtract the respective baseline variance from each calculated variance. 

Use the resulting difference in variance as the target variance for adjusting the error 

distributions. If the target variance is zero or negative, leave all error values of the 

distribution as is and do not proceed to the next step. If the target variance is positive, 

iteratively solve to find a single numerical value that can be used to divide each error in a 

given distribution such that the resulting distribution has a variance equal to the target 

variance. Now each of the errors is corrected for baseline variance. 

38
 
 

Then, calculate the NTE result with all errors, including torque and flow errors set to zero. 

This is the true value. Then subtract the true NTE value from the result with all errors 

and record this difference in one of the 7 measurement allowance distributions: mPM 

times three calculation methods (torque-speed, fuel-specific * BSFC, ECM fuel flow) 

times three PEMS manufacturers, except Sensors and Horiba can not use the ECM fuel 

flow calculation method. Then proceed to the next NTE event in the nominal data set. 

Repeat the entire nominal data set over and over until all 7 measurement allowance 

distributions converge. Follow the data reduction steps set out in Section 2 to select the 

final measurement allowance. 

4.2 PM Generator Commissioning 

The PM generator is developed by EPA. The PM generator can create various 

hydrocarbon mixtures along with solid particle generation using carbon rods arcing. The 

PM generator is also equipped with a micro-proportional diluter, and is intended to 

simulate diesel exhaust particle phase compounds. 

EPA will ship the PM generator to SwRI. EPA (Matt Spears) will train SwRI staff on 

using it. In addition, SwRI together with EPA may incorporate to it a soot particle 

generation mechanism that is different than the carbon rod arcing, using instead a 

propane flame mini-CAST technology. 

The PM generator will be used during atmospheric chamber testing, temperature and 

humidity testing, and may be used during EMI/RFI and vibration experiments. 

4.3 Baseline 

4.3.1 Objective 

The baseline variance will be established using an 8 hour baseline test in which the PM 

generator cycles through the same compositions and concentrations of PM used during 

the actual environmental tests. Mean values will be determined from the first five cycles 

through the PM concentrations. Deviations (deltas) from these mean values during 

subsequent cycles through the concentrations will be used to determine the baseline 

variance. This variance will be subtracted from the environmental test results. 

4.3.2 Background 

All of the other environmental tests inherently incorporate the baseline bias variance of 

the PEMS. Because the Monte Carlo simulation model adds all the errors determined 

from the various environmental tests, it would add the baseline variance of PEMS to the 

model too many times. In order to compensate for this in the model, the baseline 

variance of PEMS is determined and subtracted from each of the environmental tests’ 

results. 

Note that the baseline variance of PEMS is measured and modeled (i.e. added) once as 

part of the steady-state engine dynamometer laboratory experiment. 

39
 
4.3.3 Methods and Materials 

For this experiment use a well ventilated EMI/RFI shielded room capable of maintaining 

reasonably constant temperature and pressure. Use a room that can house one of each 

PEMS, their power supplies, the PEMS flow meters, cables and lines. 

Prior to executing the baseline test, setup each PEMS and stabilize the PEMS in the room. 

Perform PEMS setup according to 40 CFR Part 1065 Subpart J and PEMS manufacturer 

instructions, including any warm-up time, and audit. Then supply the PEMS’ sample 

ports with the sequence of PM from the PM generator as described at the beginning of 

Section 4. 

At each PM concentration, flow PM long enough so that stable readings of the PEMS can 

be recorded. When the OC/EC analyzer is used to spot-check the output of the PM 

generator, ensure that enough time has elapsed to achieve an accurate OC/EC analysis. 

Position PEMS and configure PM transport tubing to minimize transport delays and PM 

losses. 

Test at least one PEMS from each PEMS manufacturer. 

4.3.4 Data Analysis 

Reduce the baseline data for each PM PEMS, using artificial NTE sampling event times. 

Subtract from each mPM the mean mPM from the initial (short) baseline test of six cycles 

through the PM concentrations, which were conducted at the beginning of the test. The 

results are errors or “deltas”. Calculate the variance of these values, and use them for 

baseline variance correction in the data reduction of the remaining environmental tests. 

4.4 Electromagnetic Radiation 

4.4.1 Objective 

Evaluate the effect of Electromagnetic Interference (EMI) and Radio frequency 

Interference (RFI) on the performance of the PEMS and determine error factors for the 

PEMS due to these effects. First, a screening test on each instrument will be performed 

with HEPA filtered air to determine if the EMI/RFI affects the instrument response. If it 

does, the MASC will decide on the test matrix required for this evaluation. 

4.4.2 Methods and Materials 

Use an EMI test facility capable of running the SAE tests listed above. This would 

include: Signal generators, Power amplifiers, Transmit antennas, Electric Field Sensors, 

Measurement Receiver, Data recording device, LISNs (Line Impedance Stabilization 

Networks) and shielded enclosure. 

40
 
 

4.5 Atmospheric Pressure 

4.5.1 Objective 

Evaluate the effects of ambient pressure on PEMS PM concentration outputs. 

4.5.2 Background 

PEMS are expected to operate over ranges of ambient pressures. It is hypothesized that 

some of the errors of the PEMS concentration outputs may be a function of ambient 

pressure. Therefore, this experiment will change the ambient pressure surrounding 

PEMS to evaluate its effects on PEMS measured concentrations and flow meter 

transducer outputs. As with all of the environmental tests, the test cycle for this test is 

based on the best-known distribution of real world conditions. For this test, the test cycle 

pressure distribution was matched to the county-by-county annual average atmospheric 

pressure distribution in EPA’s 2002 National Emissions Inventory (NEI) model. Figure 

10 depicts the NEI data distribution (based on 3149 data points) and the test cycle 

pressure distribution. 

Pressure Histograms 

0% 

10% 

20% 

30% 

40% 

50% 

60% 

70% 

80% 

90% 

100% 

84.1 86.8 89.6 92.3 95.0 97.8 100.5 

Pressure (kPa) 

Frequency (% of total time) 

2002 NEI Pressure Distribution <5500 ft 

Test Cycle Pressure 

FIGURE 10. PRESSURE HISTOGRAM 

4.5.3 Methods and Materials 

Use a barometric chamber that can be well ventilated and capable of controlling a wide 

range of pressure changes (82.74 to 101.87 kPa). Use a chamber that can house at least 

41
 
 

 

 

 

 

 

 

three PEMS at a time, one of each PEMS manufacturer, their power supplies, the PEMS 

flow meters, cables and lines, plus the PM generator. 

Follow a pattern of first soaking the PEMS at a constant pressure, then ramp the pressure 

to a new pressure, soak the PEMS at that new pressure, and then ramp to another pressure. 

Use the sequence of pressures and times, as shown in Table 9, to simulate a typical 

distribution of real-world pressures and changes in pressure, which are believed to be 

dominated by changes in altitude during driving in the United States. 

TABLE 9. ATMOSPHERIC PRESSURE TEST SEQUENCE 

Atmospheric Pressure Test Sequence 

Phase Pressure Time Rate Comments kPa Alt. ft. min ft/min 

1 Soak 101 89 10 0 Flat near sea-level 

2 Ramp 101-97 89-1203 20 56 Moderate hill climb from sea level 

3 Soak 97 1203 20 0 Flat at moderate elevation 

4 Ramp 97-101.87 1203- -148 60 -23 Moderate descent to below sea 

level 

5 Soak 101.87 -148 20 0 Flat at extreme low elevation 

6 Ramp 101.87-101 -148-89 20 12 Moderate hill climb to near sea 

level 

7 Soak 101 89 20 0 Flat near sea level 

8 Ramp 101-97 89-1203 20 56 Moderate hill climb from sea level 

9 Soak 97 1203 25 0 Flat at moderate elevation 

10 Ramp 97-96.6 1203-1316 20 6 Slow climb from moderate 

elevation 

11 Soak 96.6 1316 20 0 Flat at moderate elevation 

12 Ramp 96.6-82.74 1316-5501 20 209 Rapid climb to NTE limit 

13 Soak 82.74 5501 20 0 Flat at NTE limit 

14 Ramp 82.74-96.8 5501-1259 30 -141 Rapid descent from NTE limit 

15 Soak 96.8 1259 20 0 Flat at moderate elevation 

16 Ramp 96.8-90 1259-3244 15 132 Rapid hill climb to mid elevation 

17 Soak 90 3244 10 0 Flat at mid elevation 

18 Ramp 90-96.8 3244-1259 20 -99 Rapid descent within middle of 

NTE 

19 Soak 96.8 1259 20 0 Flat at moderate elevation 

20 Ramp 96.8-99.2 1259-586 20 -34 Moderate descent to lower 

elevation 

21 Soak 99.2 586 20 0 Flat at lower elevation 

22 Ramp 99.2-101 586-89 10 -50 Moderate decent to near sea-level 

23 Soak 101 89 20 0 Flat near sea-level 

42
 
______

 

Pressure-Time Environmental Test Cycle 

9000 

7000 

5000 

3000 

1000 

-1000 

______ Altitude (ft) _ _ _ _ 1/2-hr moving avg dA/dt (ft/hr) ______ Vertical gridlines = hours

 Vertical gridlines = 7-min gas

 cylinder cycle times 

71 

75 

79 

83 

87 

91 

95 

99 

103 

107 

111 

115 

119 

123 

127 

Approximate Pressure (kPa) 

Altitude (ft) & dA/dt (ft/hr) -3000 

-5000 

-7000 

-9000 131 

0 1 2 3 4 5 6 7 8 

Time (hr) 

FIGURE 11. PRESSURE-TIME ENVIRONMENTAL TEST CYCLE 

Prior to executing this pressure sequence, setup each PEMS and stabilize the PEMS in 

the chamber’s first pressure. Perform PEMS setup according to 40 CFR Part 1065 

Subpart J and PEMS manufacturer instructions, including any warm-up time, zero-spanaudits of the analyzers and the setup of all accessories including flow meters, ECM 

interpreters, etc. Then supply the PM PEMS’ sample port with the sequence of PM from 

the PM generator as described at the beginning of Section 4. 

Flow each generated PM sample long enough so that at least 30 seconds of stable 

readings are recorded for the slowest responding gas concentration output of all the 

PEMS. Position PEMS and configure gas transport tubing to minimize transport delays. 

Target to sample about 30 seconds. Repeat this cycle over the 8-hr test cycle, by cycling 

through the concentration shown in Table 8, which represents one hour of testing, using a 

4.5 minutes of stabilization and 30 seconds of sampling at each condition. 

Perform this test once for one set of PEMS with as many PEMS tested at once. 

4.5.4 Data Analysis 

Perform data analysis according to Section 4.1. 

43
 
4.6 Ambient Temperature and Humidity 

4.6.1 Objective 

Evaluate the effects of ambient temperature and humidity on PEMS PM concentration 

outputs. The histogram in Figure 12, along with Table 10 and Figure 13, will be updated 

by a new temperature profile that takes into consideration the data generated by CECERT. 

4.6.2 Background 

PEMS are expected to operate over a wide range of changing ambient temperatures. It is 

hypothesized that some of the errors of the PEMS outputs may be a function of changes 

in ambient temperature. Therefore, this experiment will change the ambient temperature 

surrounding PEMS to evaluate its effects on PEMS measured concentrations and flow 

meter transducer outputs. As with all of the environmental tests, the test cycle for this 

test is based on the best-known distribution of real world conditions. For this test, the test 

cycle temperature distribution was matched to the hour-by hour county-by-county 

average atmospheric temperature distribution, weighted by vehicle miles traveled 

according to EPA’s 2002 National Emissions Inventory (NEI) model. Figure 12 depicts 

the NEI data distribution (based on over 900,000 temperatures and over 270 trillion 

vehicle miles) and the test cycle temperature distribution. 

Temperature Histograms 

0% 

10% 

20% 

30% 

40% 

50% 

60% 

70% 

80% 

90% 

100% 

28.5 39.5 50.5 61.5 72.5 83.5 94.5 

Temperature (F) 

Frequency (% of total time) 

Real World Temperature 23 -100 F 

Test Cycle Temperature 

FIGURE 12. TEMPERATURE HISTOGRAM 

44
 
4.6.3 Methods and Materials 

Use a well ventilated room capable of controlling a wide range of temperature changes (­

23 to 100 °F). Use a room that can house at least six PEMS, their power supplies, the 

PEMS flow meters, cables and lines, plus seven different zero, audit, and span gas 

cylinders, and a gas switching system. 

Follow a pattern of first soaking the PEMS at a constant room temperature, then ramping 

the room temperature to a new temperature, soaking the PEMS at that new temperature, 

and then ramping to another temperature. Use the following sequence of temperatures, 

shown in Table 10, and times to simulate the range of real-world temperatures and 

changes in temperature: 

TABLE 10. AMBIENT TEMPERATURE TEST SEQUENCE 

Ambient Temperature Test Sequence 

Phase 

Temperature Tim 

e 

Rate 

Comments 

°C °F min °C/mi 

n 

1 Soak 13.89 57 10 0.00 Cool in-garage pre-test PEMS 

operations 

2 

Ramp 13.89-5.00 57-23 5 -3.78 Leaving cool garage into cold ambient 

3 Soak -5.00 23 5 0.00 Operating at cold temperature outside 

of vehicle 

4 

Ramp 

-5.00­

12.78 23-55 145 0.12 Diurnal warming during cool day 

5 Soak 12.78 55 40 0.00 Steady cool temperature during testing 

6 

Ramp 

12.78­

28.33 55-83 5 3.11 Return to hot garage on a cool day 

7 Soak 28.33 83 52 0.00 Hot in-garage pre- post- test PEMS 

operations 

8 

Ramp 

28.33­

37.78 83-100 5 1.89 Leaving ho garage into hot ambient 

9 Soak 37.78 100 8 0.00 Operating at hot temperature outside 

of vehicle 

10 

Ramp 

37.78­

22.22 100-72 100 -0.16 Diurnal cooling during hot day 

11 Soak 22.22 72 60 0.00 Steady moderate temperature during 

testing 

12 

Ramp 

22.22­

13.89 72-57 5 -1.67 Return to cool garage on a moderate 

day 

13 Soak 13.89 57 40 0.00 Cool in-garage post-test PEMS 

operations 

45
 
_____

Temperature-Time Environmental Test Cycle 

-60 

-50 

-40 

-30 

-20 

-10 

0 

10 

20 

30 

40 

50 

0 1 2 3 4 5 6 7 8 

Time (hr) 

Temperature (C) 

-76 

-64 

-52 

-40 

-28 

-16 

-4 

8 

20 

32 

44 

56 

68 

80 

92 

104 

116 

Temperature (F) 

_____ Temperature _ _ _ _ 1/2-hr moving average dT/dt (C/hr) _____ Vertical gridlines = hours

 Vertical gridlines = 7-min gas cylinder cycle times 

FIGURE 13. TIME SERIES CHART OF AMBIENT TEMPERATURE TEST 

Prior to executing this temperature sequence, setup each PEMS and stabilize the PEMS in 

the chamber’s first temperature. Perform PEMS setup according to 40 CFR Part 1065 

Subpart J and PEMS manufacturer instructions, including any warm-up time, zero-spanaudits of the analyzers and the setup of all accessories including flow meters, ECM 

interpreters, etc. 

Run the 8-hour cycle test by stepping through the concentration and dilution ratio shown 

in Table 8. 

4.6.4 Data Analysis 

Perform data analysis according to Section 4.1. 

4.7 Orientation and Vibration 

4.7.1 Objective 

Evaluate the effect of vehicle vibration on the performance of the PEMS and determine 

error factors for the PEMS due to these effects. Prior to doing extensive vibration work, 

perform a screening using HEPA filtered air sampling at a sweep of different frequencies 

with low amplitude. If any of the PEMS shows a response to a particular frequency, 

propose a frequency test and submit it for the MASC for approval. 

46
 
5 SWRI CVS AND CE-CERT TRAILER CORRELATION 

Prior to performing the in-use work with the PM-PEMS, it is important to establish 

the degree of correlation between SwRI CVS-based PM measurement and CE-CERT 

CVS-based PM measurement. For this purpose, the CE-CERT trailer will move to 

SwRI facilities and PM measurement will be conducted on the engine used for the 

PM-PEMS program. 

5.1 Method and Materials 

Below is a list of a step by step approach for the correlation between SwRI and CECERT 

1.	 Perform a propane check on SwRI CVS and 47 mm filter and CE-CERT 

CVS and 47 mm filter. Both systems should pass Part 1065 on propane. 

However, even if they pass, note any difference between the two. 

2.	 Set the CVS flow rate to be the same on both systems 

3.	 Set the filter face temperature and velocity to be the same on both systems 

4.	 Set the secondary dilution ratio to be the same on both systems. 

5.	 Use Whatman PTFE membrane filters (7592-104), and filter screens that 

meet the latest Part 1065. 

6.	 Modify the exhaust path to SwRI CVS to be comparable with that for the CECERT Trailer 

7.	 Pre-condition the SwRI CVS tunnel and the CE-CERT trailer CVS tunnel for 

a period of 10 hours at engine rated power using exhaust configuration with 

DPF without bypass. The conditioning time may include active DPF 

regenerations. 

8.	 Run a total of 12 repeats of the NTE transient cycle using DPF with Bypass 

Level at 0.025 g/hp-hr emission level, over a period of three days. Four 

repeats per day with the CE-CERT followed by four repeats with SwRI CVS 

and then alternate. Prior to each set of four repeats manually regenerate the 

DPF. 

9.	 Use SwRI DMM-230 and CE-CERT DMM-230 to make sure that the engine 

PM source is not shifting and being consistent. 

10. SwRI should handle and weigh all the filters for both SwRI and CE-CERT in 

accordance with their protocol. 

11. The CE-CERT trailer is needed at SwRI for at least two weeks per engine. 

One week for setup and two weeks of testing assuming the above schedule. 

12. In a separate task, EPA will equilibrate and pre-weigh 20 filters using EPA’s 

weighing protocol. EPA will then ship them to SwRI for repeat preweighing 

using their protocol. SwRI will then ship the same filters to EPA for 

reweighing. After reweighing at EPA, EPA will ship the filters to CE-CERT 

for weighing using CE-CERT’s weighing protocol. Finally, CE-CERT will 

47
 
ship the filters to EPA for reweighing. Results will be reported by EPA for 

MASC discussion. No threshold for acceptance has been established at the 

time of this testplan writing. 

13. The target for correlation at the 0.025 g/hp-hr level is CE-CERT’s mean of 

12 repeats being within +/-10% of the mean value reported by SwRI. 

6	 MODEL VALIDATION AND MEASUREMENT ALLOWANCE 

DETERMINATION 

6.1 Model validation 

6.1.1 Objective 

Validate the Monte Carlo model by 

1.	 Testing the PEMS in parallel with the CE-CERT trailer 

2.	 Checking the data to see if it fits the model predicted based on the laboratory 

efforts 

6.1.1.1 CE-CERT Validation 

The difference between the PEMS results and the CE-CERT trailer results will be 

compared to the error predicted by the Monte Carlo model. To validate the Monte Carlo 

model, data must be run through the model and the model results must predict the actual 

test results within a reasonable level of accuracy. 

Validation will be based on the following procedure. For each reference NTE event, the 

Monte Carlo model will be used to generate the 5th and 95th percentiles of the simulated 

distribution of the brake-specific PM emission differences. In order to obtain simulations 

representing similar conditions to those obtained on-road, some error surfaces may need 

to be suppressed in the simulations since not all of them may be applicable to the on-road 

conditions. The choice of which error surfaces to suppress would need to be made by the 

Steering Committee. 

Next, the 5th and 95th delta percentiles obtained from the above simulations will be 

separately fit to a line or curve using two chosen methods: a linear regression procedure 

and a local regression (loess) technique1

. Depending on which of the resulting two fits is 

best for each set of data (i.e., either for the 5th percentile deltas or the 95th percentile 

deltas), the resulting line or curve will be used as one of the lower or upper limits for the 

on-road data. 

To determine the best fit for a given set of delta percentiles (i.e., 5th or 95th), a simple 

regression line initially will be fit to the data. If a least squares linear regression of the 5th 

or 95th percentile deltas versus the ideal PM emission has an r

2

>0.85 and an SEE < 5 % of 

the median ideal PM emissions, then the regression line will be used. If this set of 

criteria is not met, then a loess fit will be used. Since a loess regression requires the 

48
 
 

selection of a smoothing parameter2

 to smooth the data, the chosen smoothness parameter 

should balance the residual sum of squares against the smoothness of the fit. 

The on-road delta errors, obtained from the results of collecting data on several NTE 

events during on-road operations, will be plotted on a graph containing the 5th and 95th 

percentile delta limits determined from the regression fits chosen above. The graph will 

consist of a plot of delta PM versus ideal PM. The number of on-road points outside 

these limits will be determined and expressed as a percentage of the total number on onroad data points. If this number does not exceeds 10% of the total number of on-road data, 

the simulation data will be considered to be valid. 

6.2 Measurement Allowance Determination 

6.2.1 Objective 

Use the Monte Carlo simulation program developed with data from sections 2, 3 and 5, 

and validated with section 5.1 to determine the measurement allowance for all regulated 

emissions, at 2007 emissions standards. 

6.2.2 Background 

After the Monte Carlo model has been validated and confidence in its ability to predict 

errors from PEMS instrumentation, the last step in this program will be to actually 

calculate a single set of measurement allowance for PM. 

6.2.3 Methods and Materials 

Using the criteria explained in section 2.2 calculate the various levels of measurement 

accuracy corresponding to the three PEMS manufacturers and the brake specific PM 

emissions calculations. Use all the various error surfaces developed during this test 

program, including those provided by engine manufacturers to the EPA and ARB. 

6.2.4 Data Analysis 

Use the methodology explained in section 2.2, and Table 2.2 to arrive at the final 

measurement allowance. 

49
 
7 TIME AND COST 

7.1 Timeline 

Table is a tentative timeline projecting the major tasks to be accomplished during this program. The additional 

is the work that may need to be done if the model did not validate. Otherwise, the final report will be submit

2009. 

TABLE 11. PROJECTED PM-PEMS TIMELINE 
 

 

 

 

 

 

7.2 Cost 

The rough estimated cost is shown in Table 12. Based on the current estimate, a $125,000 

of the $200,000 is needed to complete the project. 

TABLE 12. PROJECTED COST ESTIMATE 

PEMS Training, Setup, Audit, and Debug $660,000 

Steady-State and Transient Experiments $190,000.00 

SwRI and CE-CERT Correlation (1 engine) $75,000.00 

PM Generator and Environmental Testing Activities $200,000.00 

Modeling Activities (Including CO2) $225,000.00 

Data and analysis, reporting, and final report $150,000.00 

Contingency if validation fails $100,000.00 

General Contingency $200,000.00 

Grand Total $1,800,000.00 

Grand Total Without General Contingency $1,600,000.00 

Grand Total without General Contingency and 

Contingency if Validation Fails $1,500,000.00 

8 ABBREVIATIONS USED IN BRAKE SPECIFIC EQUATIONS 

Method 1: 

ePM = brake-specific emission, PM (g/hp-hr) 

N = total number (of time intervals) in series 

x = amount of substance fraction (mol PM/mol exhaust; note that 1mol (emission
 

constituent)/mol (exhaust) = 1ppm (part per million) 

n

. 

= amount of substance rate (mol/sec, in this case, mol (exhaust)/sec 

�t = time interval (sec) 

fn = rotational frequency (shaft), rev/min 

T = torque (N-m) 

NOTE: The units of the numerator work out to gemission as is. However, using the 

units given for the denominator (RPM * N-m * s), you would still need to divide by 

1.978 to get to hp-hr (using RPM * N-m = kW * 9550, 1 hour = 3600 sec, and kW = 

hp*0.7457) 
Method 2: 

ePM = brake-specific emission, PM (g/hp-hr) 

MNO2 = Molecular weight, NO2 (~46 g/mol) 

N = total number (of time intervals) in series 

x = amount of substance fraction (mol PM/mol exhaust; note that 1mol (emission
 

constituent)/mol (exhaust) = 1ppm (part per million) 

~ 

n

. 

= amount of substance rate (mol/sec, in this case, mol (exhaust)/sec) that is linearly 

proportional to n

. 

(Note: this is a proportional sample, which means that you may use a 

flow meter that has a span error, as long as its calibration is linear) 

�t = time interval (sec) 

MC = Atomic weight of carbon (~12 g/mol) 

wfuel = g (carbon)/g (fuel); Note fuel is roughly 86% carbon by mass 

xCproddry = amount of carbon products on a C1 basis per dry mol of measured flow 

(exhaust), mol/mol, solved iteratively per 1065.655 

xH2O = amount of water in measured flow, mol/mol (see 1065.645 for calculations) 

efuel = brake-specific fuel consumption (g (fuel)/hp-hr) 

Method 3: 

ePM = brake-specific emission, PM (g/hp-hr) 

MNO2 = Molecular weight, NO2 (~46 g/mol) 

wfuel = g (carbon)/g (fuel); Note fuel is roughly 86% carbon by mass 

MC = Atomic weight of carbon (~12 g/mol) 

N = total number (of time intervals) in series 

x = amount of substance fraction (mol PM/mol exhaust; note that 1mol (emission
 

constituent)/mol (exhaust) = 1ppm (part per million) 

m

. 

fuel = mass rate of fuel (g/sec) 

xH2O = amount of water in measured flow, mol/mol (see 1065.645 for calculations) 

xCproddry = amount of carbon products on a C1 basis per dry mol of measured flow 

(exhaust), mol/mol 

�t = time interval (sec) 

fn = rotational frequency (shaft), rev/min 

T = torque (N-m) 

�t = time interval (sec) 

NOTE: The units of the numerator work out to gemission as is. However, using the
 

units given for the denominator (RPM * N-m * s), you would still need to divide by 

1.978 to get to hp-hr (using RPM * N-m = kW * 9550, 1 hour = 3600 sec, and kW =
 

hp*0.7457) 

52