Sierra Instruments Model BG-3 vs. CVS Multiple Engine Correlation Study 

Forward:

In 2006, Sierra Instruments Inc. recognized that demonstration of correlation of the 

Model BG-3 transient partial flow sampling system to full-flow CVS systems for 

particulate matter (PM) sampling and gravimetric measurement, per Parts 86, 1065, et al.,

was a necessary step toward acceptance of the unit as a certification-grade device. With 

input from technical partner Caterpillar Inc., a series of correlation tests were defined. In 

all, 120 pairs of simultaneously obtained test data were taken at West Virginia University 

(WVU) and at Caterpillar Inc in Part 1065-compliant CVS-equipped test cells. The

testing was conducted between March, 2008 and April, 2009.

The engines and after-treatment systems used are shown below in Table 1. Testing was 

performed on both on-highway and non-road engines on the heavy-duty FTP or non-road 

transient cycle (NRTC) as required for certification of the engine types tested. In order to 

provide a representative mix of EGR and non-EGR equipped engines from various 

manufacturers, engines from John Deere, Cummins and Caterpillar were tested, 

producing cycle PM levels ranging from approximately 0.5 to 100 milligrams/hp-hr. 

Manufacturer Model Test 

Lab

Use Cycle PM 

Limit

Emissions abatement

Cummins ISX 

565

WVU On-road FTP 2004 EGR

Deere 6081 WVU Off-road NRTC Tier 

IV

CCRT (added for test)

Cat C-11 Cat On-road FTP 2007 CCRT

Cat C-9 Cat Off-road NRTC Tier 

IV

CCRT (added for test)

Cat C-9 Cat Off-road NRTC N/A CCRC w/ Bypass

(added for test)

Cat C-13 Cat On-Road FTP 2007 Production DPF

Table 1, Test Plan Summary for BG-3 –CVS Correlation Study 

The Deere engine was a Tier II engine equipped with CCRT to achieve Tier IV NRTC 

cycle PM limits. The Caterpillar C-11 was a MY2004 engine equipped with a CCRT and

which is currently the heavy duty diesel industry round-robin test engine. The Caterpillar 

C-9 engine is used as a mule by Caterpillar for test measurement method development at 

their engine development center. The C-9 was also equipped with CCRT technology and 

included an engineered bypass for a portion of the tests. The bypass was added 

specifically to evaluate the correlation of the BG-3 to the CVS at emissions levels 

approaching the 2007/2010 on-highway engine PM emissions limits.

The testing included a 10x repetition of the heavy duty diesel engine transient cycle 

certification procedure. One cold and one hot test cycle were obtained per day for a 

minimum of 10 cold-hot repeats per engine. These results were weighted as per 40CFR 
Part 86 requirements for engine certification (1/7*Cold Result, g/hp-hr + 6/7*Hot Result, 

g/hp-hr = Weighted Cycle Emissions. g/hp-hr). Paired-t and F-test statistical analysis 

was subsequently performed on the weighted test results to determine acceptability at 

90% confidence level per Part 1065.12. Maintaining consistency between BG-3 and 

CVS specific to critical sampling system setup parameters such as filter face velocity, 

dilution ratio, residence time, filter temperature and dilution air temperature was actively 

pursued to maintain the technical integrity of the correlation study. All aforementioned 

parameters were maintained within 40CFR Part 1065-required limits.

Correlation Testing Results

The initial phase of testing was performed at the West Virginia University Center for 

Alternate Fuels, Engines and Emissions, Morgantown, West Virginia (WVU). Graphs #1 

and #2, below, denote cold-hot cycle-weighted results on the Cummins and Deere 

engines, respectively for run days 1-10. It should be noted that post-test analysis of the 

Cummins engine day #5 gaseous emissions results showed significant within-day 

increases in the hot test cycle hydrocarbon (+26.2%) and carbon monoxide (+3.7%)

emissions (directional indicators for PM), relative to results from Day 4 and Day 6, that 

were reflected in the BG-3 results (+12.7%) but not from the CVS (+1.37%). A 

weighing error specific to the CVS filter is suspected. This analysis questions only the 

CVS result from Day 5 of this series. The graph below includes the Day 5 results for 

completeness. All engines in this program were able to complete their prescribed test 

cycles in accordance with the speed and torque regression statistics required by Part 1065 

and Part 86. BG-3 Standard Error of Estimate (%SEE) results were well below the 3.5% 

limit required by Part 1065 for all runs at both WVU and at Caterpillar.

BG-3 vs CVS Cold-Hot Results, Cummins Engine, FTP Cycle

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10

Run Set #

PM, mg/hp-hr

CVS, mg/hp-hr

BG-3, mg/hp-hr

Graph 1- BG-3 vs. CVS Correlation Results, Cummins ISX 565 On-Highway Engine, 

WVU Data
BG-3 vs CVS Cold-Hot Results, Deere Tier II Engine w/ DPF, NRTC Cycle

0

0.5

1

1.5

2

2.5

3

1 2 3 4 5 6 7 8 9 10

Run Pair #

PM, mg/hp-hr

CVS, mg/hp-hr

BG-3, mg/hp-hr

Graph 2- BG-3 vs. CVS Correlation Results, Deere 8.1L Engine w/ CCRT, WVU Data

The second phase of correlation testing was performed at the Caterpillar Technical 

Center, Mossville, Illinois, in a CVS test cell with a fully 1065 compliant particulate 

sampling system. The C-11 industry (EPA-EMA) round robin engine with CCRT was at 

Caterpillar for inter-lab comparison testing at that time. It was considered an excellent 

candidate for BG-3-CVS correlation testing due to the number of labs having already 

tested the engine, including EPA. Graph 3 shows the cold-hot weighted results from the 

C-11 tests at Caterpillar. The variability in results shown in Graphs 2, 3, 4 and 6 are

dominated by uncertainties in the filter weighing process, coupled with engine and 

CCRT-based day-to-day variability. This level of variability is an expected outcome 

when measuring PM, using approved gravimetric methods, from engines operating at 

between 5 and 20% of the 2007 PM standard. There were no emissions system or engine 

setup changes made during the course of the test and test filters were re-weighed to verify 

PM mass results. 

Table 2 compares BG-3 and Caterpillar CVS data against results obtained by participants 

in the EPA-EMA round-robin inter-lab comparison program. This chart includes the 

coefficient of variance (1std dev/mean) relative to the 2007/2010 particulate standard. 

This comparison underscored the correlation for both sample mean and variance values 

from the Caterpillar CVS and BG-3 versus a broader population of test labs from testing 

a common engine and after-treatment package. 
 BG-3 vs. CVS, Cold-Hot Weighted Results, C-11 R-R Engine, FTP Cycle 

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1 2 3 4 5 6 7 8 9 10

Run #

PM, mg/hp-hr

CVS, mg/hp-hr

BG-3, mg/hp-hr

Graph 3- BG-3 vs. CVS Correlation Results, C-11 Engine w/ CCRT, Caterpillar Data

Lab Cold/Hot 

PM, g/hphr

COV Expressed 

as % of 2010 PM 

limit 

A .0012 7.0

B .0007 1.0

C .0010 3.0

D* .003 10.13

A2** .0031 20.2

E .0007 4.49

F .0007 2.08

Cat CVS .0010 4.58

BG-3 .00115 7.66

*-Lab used incorrect filter media

**Reran using denormalized FTP test cycle; labs listed after lab A2 ran the denormalized 

cycle

Table 2- Comparison of BG-3 and Caterpillar CVS mean and % coefficient of variance 

(COV) results with test labs participating in the C-11 round-robin engine study.
 BG-3 vs. CVS Cold-Hot Weighted Results, Tier III C-9 w/ DPF

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

2.00

1 2 3 4 5 6 7 8 9 10

Run #

PM, mg/hp-hr

CVS, mg/hp-hr

BG-3, mg/hp-hr

Graph 4- BG-3 vs. CVS Correlation Results, C-9 Engine w/ CCRT, Caterpillar Data

During testing, questions regarding the BG-3 to CVS correlation at PM levels 

approaching the 2007/2010 standard were raised, essentially due to the disparity in PM 

levels generated by the selected engines relative to both the on-highway 2007/2010 and 

non-road Tier IV emissions standards. The results found on Graph 5 were obtained 

during BG-3 to CVS correlation testing from a C-9 with an engineered 17.25mm fixed 

bypass installed around the CCRT, selected to yield an emissions level of approximately 

85% of the 2007/2010 standard. The engine was operated on the NRTC for this 

correlation exercise. Comparing the paired results from this test sequence showed that 9 

out of 10 cold-hot BG-3 results are within less than 10% of the corresponding CVS 

result. Reviewing this same paired dataset using a tighter comparison standard, 8 out of 

10 of the cold-hot BG-3 results were within 6% of their CVS cold-hot cohorts. This 

agreement underscores the correlation integrity across the anticipated range of PM levels 

to be certified (or re-certified) using Part 86 or Part 1065 methods on NRTC or FTP 

cycles.

Graph 6 summarized cold-hot results obtained on the 2008 Caterpillar C-13 engine. As 

previously stated, elevated levels of variability from both measurement systems are 

expected when sampling particulate matter at 5% of the 2007/2010 PM emissions limit. 

In the case of the C-13, net filter loadings of less than 5 micrograms were routinely 

measured from both the CVS and BG-3.
BG-3 vs. CVS Cold-Hot Weighted Results, C-9 with Bypassed DPF, NRTC

0.00

2.00

4.00

6.00

8.00

10.00

12.00

1 2 3 4 5 6 7 8 9 10

Run Number

PM, mg/hp-hr

CVS, mg/hp-hr

BG-3, mg/hp-hr

Graph 5- BG-3 vs. CVS Correlation Results, C-9 Engine w/ partially bypassed CCRT, 

Caterpillar Data

BG-3 vs. CVS Cold-Hot Weighted Results, C-13 Engine, FTP Cycle 

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1 2 3 4 5 6 7 8 9 10

Run #

PM, mg/hp-hr

CVS, mg/hp-hr

BG-3, mg/hp-hr

Graph 6- BG-3 vs. CVS Correlation Results, 2008 C-13 Engine, Caterpillar Data
In conclusion, based on test results from this initiative, the Sierra BG-3 passes paired-t 

and F-test results for equivalent means and equivalent variances, respectively for cold-hot 

cycle weighted transient test results per Part 1065.12, on engines ranging from 

approximately 1/20 to 10x the applicable limit for 2007 on-road and Tier IV off-road 

engine certifications. Further, based on test results from an engine with a CCRT and 

engineered partial bypass, the Sierra BG-3 passes paired-t and F-test results for 

equivalent means and equivalent variances, respectively for cold-hot cycle weighted 

transient test results per Part 1065.12 at PM emissions levels of approximately 85% of 

the applicable limit for 2010 on-road, and approximately 60% of the Tier IV off-road 

standard.