RTI International is a trade name of Research Triangle Institute



# Peer Review of LDV/ LDT OBD and
High Evaporative Emission Report

Work Assignment 4

- 0 1

(RTI 00 5 )

Technical Memorandum

Prepared for

Kent Helmer & Constance Hart
U.S. Environmental Protection Agency Office of Transportation and Air Quality (OTAQ)

Washington, DC 20460

Prepared by

Dileep K. Birur & Tony Lentz

RTI International Research Triangle Park, NC 27709

EPA Contract Number EP

- C

- 08

- 008

RTI Project Number 0211577.00 5 .00 1

May 2012Kent Helmer, Constance Hart

May 8, 2012 Page 2



### Technical Memorandum on Peer Review of LDV / LDT OBD

and High Evaporative Emission Report

Table of Contents

Contents Page
Background

................................ ................................ ................................ ............ 1

- 1
Description of Review Process

................................ ................................ ............ 2

- 2
Summary of Review Comments

................................ ................................ ............ 3

- 8
Appendix

- A: Resumes of Selected Reviewers

................................ ............... A1

- A 1 1
Appendix

- B: Charge Questions

................................ ................................ ........ B1

- B 4
Appendix

- C: Questions and Answers

................................ ................................ ..... C1
Appendix

- D: Cover Letters

................................ ................................ ............... D1

- D 3
Appendix

- E: Review Reports

................................ ................................ ......... E1

- E 11RTI International is a trade name of Research Triangle Institute

TO: Kent Helmer and Constance Hart

, U.S. Environmental Protection Agency

, Office

of Transportation and Air Quality (OTAQ)

FROM: Dileep K. Birur

, Tony Lentz, RTI International.

DATE: May 8 , 201 2

.

SUBJECT: Peer Review of Eastern Research Group’s ( ERG ’s) “Evaluation of the

Effectiveness of On

- Board Diagnostic (OBD) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles .”



### 1. Background

The EPA’s Office of Transportation and Air Quality (OTAQ) has recently commissioned
a report, Evaluation of the Effectiveness of On

- Board Diagnostic (OBD) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles , analyzing high evaporative (evap) emissions field data reported for light

- duty vehicles/trucks collected through Denver Inspection and Maintenance (I/M) lanes. This study involves comparison of information on OBD evaporative system diagnostic trouble codes (DTCs) with Sealed Housing Evapo rative Determination ( SHED ) results for the same vehicles. The high evaporation emission data used in the study were collected over the past 4 years in the Denver, Colorado area using laboratory and portable SHED units and remote sensing devices (RSD) on m ost 1996 and later vehicles. Data were collected in 3 different studies: Lipan Participant (LP), Ken Caryl Participant (CP), and Denver 2010 Participant (DP). The information gathered in these “high

- evap” studies allows for a comparison
to OBD data which were either collected at I/M Stations or in the CDPHE laboratories.

. The main purpose of the ERG’s analysis was to evaluate OBD’s ability to identify vehicles with elevated evap orative emission levels. The analysis indicated that many vehicles with high SHED values ( hot

- soak emissions ≥ 1 gram per quarter hour) do not have stored evaporative DTCs. and were therefore not identified by the OBD evaporative system.

In support of EPA’s Tier 3 rulemaking, EPA has sought the review ers’ expert opinion on its contractor’s (ERG ’s ) report of evaporative emission field data and related OBD results for light

- duty passenger cars and trucks, particularly on the methodology, and the validity of the data and assumptions that go into this anal ysis. RTI International facilitated this peer review and this technical memorandum contains documentation of the peer review process of the ERG’s study.Kent Helmer, Constance Hart

May 8, 2012 Page 2



### 2. Description of Review Process

RTI began the review process on Janua ry 28, 2012, which continued for a period of approximately 2 months. EPA provided a non

- comprehensive list of subject matter experts from academia and industry (Appendix B of the Statement of Work, WA 4

- 01) to RTI, and this served as a “starting point” from which we assembled the list of subject matter experts. To ensure that the review process was performed in a timely manner, RTI contacted the potential reviewers within ten days of submitting the work plan and determined that each expert would be able to review the study during the peri od of performance. RTI selected three independent (as defined in Sections 1.2.6 and 1.2.7 of EPA’s Peer Review Handbook) subject matter experts based on their expertise and their interest to perform the review in the stipulated time frame. In order to

ma ke the review process as credible as possible, RTI did not consult the EPA in the final selection of the reviewers. Appendix A of this technical memorandum provides the resumes
obtained from the selected reviewers.

RTI provided the peer reviewers with t he ERG report “Evaluation of the Effectiveness of
On

- Board Diagnostic (OBD) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles” via email on February 19, 2012. In addition, the peer reviewers were also given a “charge letter” prepared by the EPA. The note from RTI sent to the reviewers with the charge letter is included in Appendix B of this memorandum.

Upon distributing the review material and charge letter to the peer reviewers, RTI advised the peer reviewers to formulate any questions or concerns regarding the review material or charge letter in the form of an email correspondence. This process provided the peer reviewers the opportunity to discuss any questions or concerns they may have regarding the review material provided and the expe cted deliverables. Any reviewer submitted questions/comments/concerns were promptly f orwarded to the EPA

- WAM and Alternate WAM via email. Some of the questions addressed in this process and the answers provided are included in Appendix C of this memorandu m.
RTI received two completed reviews from the peer reviewers and forwarded on to the EPA by the requested date, March 9, 2012. The third review report was submitted to RTI and forwarded on to the EPA on March 20, 2012. The review reports included resp onses to the charge letter and any additional comments or recommendations the reviewers may have had. From each review participant, RTI obtained a cover letter stating the reviewer’s name, the name and address of their organization if applicable, which rev iew documents/media were received by the reviewer and which was actually reviewed and a statement of any real or perc eived conflict(s) of interest. These cover letters and the review reports are included in Appendices D and E, respectively, of this memoran dum.Kent Helmer, Constance Hart

May 8, 2012 Page 3



### 3. Summary of Review Comments

ERG recently completed its analysis on “ Evaluation of the Effectiveness of On

- Board Diagnostic (OBD) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles

, ” on evaluating the effectiveness of Second Generation On

- Board Diagnostic (OBD) systems through a comparative analysis of evaporative emissions field data (e.g. Lipan Participant (LP), Ken Caryl Participant (CP), and Denver 2010 Participant (DP)) collected in the Denver area during the past four years. The focus of the ERG report was an analysis of evaporative emissions and related data collected on model year 1996 and newer light

- duty vehic les equipped with OBD systems, specifically to evaluate OBD’s abil ity to identify vehicles with elevated evaporative emissions. Three types of data were used in the analysis: On

- board Diagnostic (OBD) codes, Modified California Method (MCM) inspection, and Laboratory Sealed Housing for Evaporative Determination (LSHED) or a Portable Sealed Housing for Evaporative Determination (PSHED). The key finding of the ERG report was that a large number of vehicles with high evaporative emissions were not identified by the OBD evaporative system. Additionally, the ERG report

rec ommends since the Colorado Department of Public Health and Environment has been performing ongoing studies in this area, there could be opportunities in the future to leverage their work into further improvement of the preliminary correlations developed in the study.

The rest of this section gives a summary of the review comments received from the three peer reviewers: Rob Klausmeier (de la Torre Klausmeier Consulting), Ste phen A. Leydon (Georgia Department of Natural Resources, Air Protection Branch), an d Stephen J. Stewart (Pacific Vehicle Testing Technologies).



### 3.1 An Overview of the Reviews All three reviewers addressed the report’s correlation between results of portable SHED tests and the presence of OBD diagnostic trouble codes. One reviewer comme nted that the prescreening analysis carried out using RSD tests was inadequate and therefore, the conclusions made in the report are not valid. The other two reviews noted in similar fashion the potential shortcomings of the RSD prescreening analysis (add itional discussion of the prescreening analysis issues are provided in subsection 3.2.1 below) , but instead, noted that the RSD prescreen, however limited in its ability, provided some validation of the conclusions presented in the ERG report. Those revie wers, who felt the prescreening analysis did not jeopardize the overall conclusion of the ERG report, suggested that the analysis could be beneficial for future evaporative emissions analysis. Specifically, Mr. Klausmeier comment ed, “ the data collected

an d analyzed in this study are inadequate to make these conclusions, and that the report should not be released until sampling concerns are addressed. ” However, Mr. Leydon stated “the ERG study also indicates that many OBD II equipped vehicles are emitting a t levels identifiable by RSD and verifiable using SHED testing. ” He further indicated that the study provides valuableKent Helmer, Constance Hart

May 8, 2012 Page 4

data regarding the ability of OBD II in identifying the evaporative issues with vehicles. The third reviewer, Dr. Stewart expressed simi lar view that , since t he pre

- screening was ineff ective, it can be concluded that the evaporative DTCs are very unlikely to be present if SHED results are low.



### 3. 2 Methodology

This section highlights some of the key issues raised by the review panel regarding the methodology employed in the ERG study. On selection of vehicles with “high evap” emissions, a prescreening test using a remote sensing device (RSD) was employed to ide ntify vehicles with potentially high evaporative emissions. Irrespective of their final conclusions, all the three reviewers expressed their concerns on the potential shortcomings of the RSD prescreening analysis. B eyond the questionable RSD prescreen, th e reviewers either had no comments regarding the methodology or minor comments and/or clarifications.

3.2.1 Prescreening

Even t hough the RSD prescreening was not the primary focus of the ERG report, all the
three reviewers indicated concerns with the pres creening analysis

. S ince this was a common observation by the three reviewers, their comments are highlighted here. Mr. Klausmeier stated

,

“ The report should provide details on the number of vehicles screened by RSD, the fraction of the fleet that had high evaporative indices based on RSD, and the ultimate fraction of the fleet
that received PSHED and OBD tests.” He further believes that the conclusions of the report cannot be supported by the limited amount of data; instead “the report can only conclude O BD
misses 50

- 70% of the high evaporative emitters as determined by the RSD evaporative index, with the sample size caveat.” However, Mr. Leydon ass esse d the potential shortcomings of RSD analysis differently, remarking “... (detection by RSD) is a positive f inding for programs that continue to perform RSD for program evaluation. It would lend credibility to RSD findings of “high emitters” regardless of the results of paired OBD records. This finding provides the reason why OBD II equipped vehicles that are flagged as gross polluters by RSD will pass an OBD II inspection. This is something we have seen in Georgia in past years. ” With the similar inference,
Dr. Stewart note d “ The benefit of this ineffective pre

- s creen is that the sample did in fact include

a large number of vehicles that did not have high evaporative emissions, and it is nice to see that they also had (almost) no evaporative DTCs. ”



### 3.3 OBD

The On

- Board Diagnostic (OBD) codes indicate system or component malfunctions. The second generation OB D (OBD II) system is used to detect the malfunctions that could result in emissions above a specified threshold level. In general, each of the three reviewer s commentedKent Helmer, Constance Hart

May 8, 2012 Page 5

on the OBD systems’ abilities to accurately identify evaporative emission leaks. It should be noted that each reviewer provided varying comment ( s ) with respect to the OBD systems’ capabilities to identify evaporative emissions, while all three reviewers agreed that OBD systems cannot accurately identify all sources of evaporative emission s. Mr. Klausmeier stated “ The report should attempt to compare the distribution of diagnostic trouble codes (DTCs) observed in this sample with the distribution of DTCs found in vehicles tested in I/M programs .”

Whereas, concerning the failure of some early model year OBD systems to correctly identify an evaporative emission point, Mr. Leydon commented “The OBD systems in these four early OBD II vehicles did not store an evaporative DTC because they do not have en hanced evaporative monitoring. This finding may result in non

- enhanced evaporative monitored vehicles being given more weight as polluters in modeling due to their demonstrated lack of being able to identify high evaporative emissions or even the presence of a fuel cap at all.” He f urther indicated that the OBD tests are valid particularly in the first two studies, since OBD was evaluated shortly before the SHED test (or the previous day) and he believe s that very few vehicles change their emissions status over night.

Similarly, Dr. Stewart pointed out that though SHED results indicate the total rate of evaporative emissions, they cannot detect the source of emissions on the vehicle. He further commented that, in contrast, an OBD system tests only specific aspects of the vehicle hardware and control system while ignoring all other possible sources of evaporative emissions.



### 3.4 SHED Results

Overall, each reviewer commented on the Sealed Housing for Evaporative Determination (SHED) tests performed in each o f the three studies. In general, there were few concerns amongst the reviewers concerning the results of the SHED tests. One reviewer felt that “t he report should present more details on how vehicles were procured for the PSHED test. ” A dding
“i nformatio n on the year, make and model of vehicles that received PSHED tests would be
useful. ” Mr. Leydon commented on the SHED test results stating “ t he establishment of the PSHED at the Denver facility and correlating the readings to LSHED ensured that, for all

p ractical purposes the readings are to be considered the same and was also a great benefit for the continuity of the study. ”

Dr. Stewart note d that certain evaporative leaks identified by SHED tests will never be identified by OBD. This is because OBD syst ems are limited in identifying vapor leaks of a
certain diameter size, while SHED tests do not suffer from this limi ted ability. He noted , “ There are only two types of evaporative problem that OBD does detect: basic evaporative OBD only
checks the caniste r purge solenoid and flow rate; and enhanced evaporative OBD also looks forKent Helmer, Constance Hart

May 8, 2012 Page 6

vapour leaks equivalent to a certain size of hole. So high SHED results caused by liquid leaks or tailpipe residuals could never be detected by current OBD evaporative monitors. ”



### 3. 5 Modified California Method (MCM)

The ERG report also analyzed the relationship between the results of three high evaporative vehicle emission field studies and t he Modified California Method. Two of the reviewers provided comments on the MCM inspection. Both reviewers believe d that the analysis would benefit from further investigation. Dr. Stewart state d “... the deductions from the MCM tests are also somewhat speculative as well as being subjective. In guessing whether the leak
should have be en detected by OBD, they also assume that the OBD system would have had the chance to detect the problem, but this may not have been the case. It would be valuable to take the investigation further using an evaporative testing cart to measure vapor leaks r ates .” Whereas, Mr. Klausmeier believe d that the authors of the ERG study should “ collect and analyze data from California roadside emission tests, and other sources to validate, if possible, the assumption that vehicles with high evaporative emissions id entified in this study are typical of all vehicles with high evaporative emissions ”



### 3. 6 OBD SHED Pairing

The central focus of the ERG report was to evaluate the effectiveness of OBD to correctly identify vehicles with evaporative emissions which were measu red with SHED tests. The ERG
report matched each vehicle in the three studies with corresponding OBD and SHED tests. Overall, the reviewers felt that the matching protocol was effective and valuable to the study’s analysis. Mr. Klausmeier was not conclu sive on supporting or opposing the OBD SHED matching protocol. Dr. Stewart’s review did not offer any specify comments on the OBD SHED matching protocols carried out in each of the three studies.

Whereas, Mr. Leydon , in discussing the effectiveness of the matching protocol for the “Denver 2010” study, note d that “This group had a 100% OBD/SHED match rate presumably because the vehicles were recruited and all tests were performed on location during one visit,
which is the ideal situation .” Mr. Leydon’s comments suggest ed that the matching protocol used for the “Denver 2010” study is valuable and should be followed in future investigations of “high evap” studies. This is reasonable assumption considering that there were, according to Mr. Leydon, “ No spec ifics are given provided for the high dropout due to no OBD II readings ” in the Lipan and Caryl studies.Kent Helmer, Constance Hart

May 8, 2012 Page 7



### 3.7 REVIEWERS’ RECOM MENDATIONS

Though there are several similarities in the comments made by the peer reviewers on the ERG study, each of the reviewers has given different recommendations and follow

- on option s

. Mr. Klausmeier offered a couple of suggestions for further investigation and follow up, both of which were presented in subsections 3.3 and 3.5. Mr Klausmeier believe d that the authors o f the report should collect and analyze data from “ from I/M programs, California roadside emission tests, and other sources to validate, if possible, the assumption that vehicles with high evaporative emissions identified in this study are typical of all v ehicles with high evaporative
emissions” as well as “ attempt to compare the distribution of diagnostic trouble codes (DTCs) observed in this sample with the distribution of DTCs found in vehicles tested in I/M programs .”

Whereas, Mr. Leydon was supportive of the ERG study results and suggested “If this study is a good indication of real life experience, and it should be, modeling of OBD II vehicles without enhanced evaporative strategies should be weighted more than newer OBD II vehicles with enhanced evap orative monitoring.” However, Dr. Stewart, regarding the prescreening analysis, suggest ed that, “efficacy of the pre

- screen was not the point of the study, and while adding additional process, it did not add anything to the value of the results.” While n ot out

right recommending to remove the current prescreening analysis, the reviewer suggest ed that the report would have benefitted by removing the initial prescreening analysis.



### 3. 8 OVERALL CONCLUSIONS

The reviews of the ERG report performed by the three reviewers indicate that , in general, that the data collected in the 3 “high evap” studies are valid and adequate enough to substantiate the conclusions drawn in the ERG report. Two of the reviewers conclude d that the ERG study’s
conclusions are well

- supported by the data, even with certain caveats, while a third reviewer
believed that the data collected are not adequate enough to support the conclusions of the report.
Mr. Klausmeier conclude d that “the data collected and analyzed in this study are ina dequate ( 10
hi g h e m itting vehicles ) to make these conclusions [conclusions from the ERG report], and that the report should not be released until sampling concerns are addressed.” Mr. Klausmeier strongly believe d that the report suffers from a number of sa mpling concerns that are centered on the RSD pre

- screening analysis.

However, Mr. Leydon conclude d “ t his study provides valuable data regarding the ability of OBD II to identify, or not, eva porative issues with vehicles... If this study is a good indicatio n of real life experience, and it should be, modeling of OBD II vehicles without enhanced evaporative strategies should be weighted more than newer OBD II vehicles with enhanced evaporative monitoring. There appears a strong correlation between vehicle ag e and emissionsKent Helmer, Constance Hart

May 8, 2012 Page 8

and between non

- enhanced evaporative strategies and newer OBD II equipped vehicles. ”

The third reviewer, Dr. Stewart stated, “ t he conclusions of the report are substantially supported by the data presented, but they do come across as being a little dismissive of the usefulness or reliability of OBD data as an input for modeling the evaporative emissions of in

use vehicles. ”

Howe ver, similar to Mr. Klausmeier, Dr. Stewart noted that the 10 high emitting vehicles that remained in table 3.5 from table 3.3 of the ERG report are “ hardly an adequate sample from which to draw compelling conclusions .” However, Dr. Stewart also noted “th ere is some suggestion that the OBD evaporative monitor might not be sensitive enough to precisely identify when a leak might cause SHED results to exceed 1.0g/Qhr, and it could be that it would only detect bigger leaks .” From this comment, it can be infe rred that the reviewer seemed to support the notion that although the sample size may be small, it is more of a function of the limited capabilities of OBD than the analysis carried out in the ERG report.

### A. Appendix A: Resumes of Selected Reviewers

Resume of Panel Participants Page

1. Mr. Rob Klausmeier A1

- A 2

2. Mr. Stephen A. Leydon A 3

- A 4

3. Dr. Stephen J. Stewart A 5

- A 11A

- 1 

### Rob Klausmeier

dKC de la Torre Klausmeier Consulting

1401 Foxtail Cove Austin, TX 78704 Telephone: 1

- (512) 447

- 3077

E

- mail: delaklaus@aol.com

Education M.B.A., Business Administration, University of Michigan, Ann Arbor, MI, 1973.

B.S.M.E., Mechanical Engineering, University of Michigan, Ann Arbor, MI, 1973.

Professional Experience
Consultant, de la Torre Klausmeier Consulting, Inc, 1994 to present.
Principal Scientist/Group Leader, Radian Corporation, Austin, TX, 1991

- 1994.
Senior Staff Engineer, Radian Corporation, Austin, TX, April 1988

- 1990.
Program Manager, Radian Corporation, Austin, TX, 1984

- 1988.
Senior Mechanical Engineer, Radian Corporation, Austin, TX, 1977

- 1984.
Refinery Engineer, Standard Oil of California, Richmond, CA, 1974

- 1976.

Co

- o p Engineer, General Motors Corporation, Oldsmo bile Division, Lansing, MI,
1967

- 1972.

Areas of Expertise Rob Klausmeier specializes in the analysis and implementation of air pollution control strategies, particularly strategies for mobile sources. In 1994, Rob formed his own consulting company, de la Torre Klausmeier Consulting, Inc (dKC). Prior to this, he was employed by Radian Corporation. During Rob's 17 years with Radian and 18 years with dKC, he has performed a range of programs with an emphasis on t he following:

Development of Air Pollution Control Strategies:

o Modeling vehicle and stationary source emissions:

 Exhaust emissions

 Evaporative emissions

o Developing control strategies for attainment of the National Ambient Air Quality Standards

o Costs and cost

- effectiveness evaluations of control strategies

o Training on Mobile Source controls Emission Controls for Large Stationary Engines:

o Identification of control strategies

o Emission reduction assessmentsA

- 2 o Cost effectiveness evaluations of controls for gas oline and diesel powered engines.

o Renewable alternatives: Wind, solar, bio

- fuels

o Integration of energy storage systems Inspection/Maintenance (I/M) programs:

o Assessment of emissions test procedures

 Identification of vehicles with excessive exhaust emissio ns

 Identification of vehicles with excessive evaporative
emissions

o Design of I/M test lanes

o Request for Proposals (RFPs) for I/M programs

o Rules and regulations for I/M programs

o Negotiation of contracts for centralized I/M programs

o Performance audits of I/M programs

o Quality Assurance (QA) Programs

- quality control, data analysis, program evaluation, acceptance tests

o Cost estimates of I/M programsA

- 3 

### Stephen A Leydon Georgia Department of Natural Resources

Air Protection Branch
Mobile and Area Sources Program 4244 International Parkway 404

- 675

- 6156 Atlanta, Georgia 30354 Steve.Leydon@dnr.state.ga.us

Education

Georgia State University Atlanta, Ga
Bachelor of Business Administratio n December 1999 Computer Information Systems

Massachusetts Bay Community College Watertown, MA
Associate in Science June 1973 Electronic Technology

Weber State University Ogden, UT
National Center for Automotive Science and Technology 2000

- present On Board Diagnostic Seminars

Colorado State University Fort Collins, CO National Center for Vehicle Emissions controls and Safety 1999

– present On Board Diagnostics and Testing Seminars

Work Experience

Certified Georgia Emissions Inspector since 1981 (Basic Program)

Georgia Dept of Natural Resources Atlanta, Georgia

Vehicle Emissions Testing Coordinator January 2000 to Present

a. Responsible for the revision and updat e of the Georgia I/M Equipment Specifications manual to meet Federal testing requirements and Georgia needs.
b. Responsible for maintaining and updating of the Rules for Enhanced Inspection and
Maintenance O.C.G.A. 391

- 3

- 20 as needed.

c. Interface with equipment manufacturers during the development of Georgia hardware and software requirements and after deployment during Phase II and Phase III of the Georgia I/M program.A

- 4

d. Work closely with the Enforcement section to develop covert auditing procedures,

processes, a nd data validation methodology.

e. Support EPD law enforcement activities such as fraudulent testing investigations and
fraudulent document identification.
f. Interface with other state and provincial agencies performing emissions testing.

g. Work with the Industry Advisory Board to resolve program issues and improve program
effectiveness.
h. Assist in the development of the Inspector Re

- certification Process for the Georgia Enhanced Emissions I/M Testing Program.
i. Perform electronic data processing for program evaluati on. j. Review annual Georgia Program Report prior to submission to EPA.
k. Review Biennial RSD Program Evaluation Report created by Georgia Tech Research
Institute under contract with EPD
l. Work with local and national Federal EPA personnel in developing approved test procedures, processes and in testing analysis.

SPX Corporation Kalamazoo, Michigan Field Service Representative Sep 1977 to Dec1999

m. Perform work and equipment training in testing programs in the following states:
C alifornia, Florida, Georgia, Illinois, Massachusetts, New Jersey, New York, Pennsylvania, Texas, and Tennessee.

n. Assist in the development of the initial Georgia inspector certification process and training as a manufacturer’s representative.

o. Instruct custo mers in the operation of State specific TSI/ASM/OBD II testing procedures.

p. Work with the State of Georgia, MCI, Parsons, and Region 4 EPA personnel on the successful completion of Phase I and Phase II of the Georgia I/M program.

q. Perform field repair of sop histicated engine diagnostic equipment including Diagnostic Equipment, Emissions Analyzers, and Computer Office Equipment.

r. Demonstrated capabilities of products to customers, salesmen, and state personnel.

s. Conducted seminars on new products and instructed new service employees in proper use, calibration, and maintenance of multiple product lines.

t. Assisted in setting up the Massachusetts program using ASM/VMAS equipment.A

- 5 

### Stephen J. Stewart Ph D., MSc., BSc., P.Eng., CEng., MRAeS Carrington Lane, Abbotsford

, Abbotsfo rd, British C olumbia, V3G 2M7.

Telephone: 1

- 604

- 556

- 8260
Email: stejste@shaw.ca

Summary

Dr. Stewart studied Aeronautical Engineering at the University of Bristol, then worked as an Airframe Systems Engineer for British Aerospace. He completed graduate work at the University of Manchester; in Mechanical Engineering for MSc, and Computer Graphics and Flight Simulation for PhD. After teaching Thermodynamics and Fluid Mecha nics at Salford for seven years he moved to Canada, and has since specialized in vehicle emissions and their control. He has extensive experience in motor vehicle emission related air quality issues and in the design, operation and evaluation of Inspection and Maintenance programs as well as other emission
control initiatives.

He has completed many projects for the British Columbia inspection and maintenance program as well as for the Ontario Ministry of Environment, and for Environment Canada. In 2004 h e travelled to Urumqi, P.R.China to help develop the air quality monitoring, motor vehicle emission control and air quality improvement project funded by the World Bank. He is an Honorary Research Associate in the Department of Statistics at the University of British
Columbia.

Dr. Stewart is a private pilot and President of the Abbotsford Flying Club, as well as a Treasurer of the Abbotsford International Airshow Society. He is the Canadian representative on the
Environment Committee of the Guild of Air Pi lots and Air Navigators and a member of the
North American Section Board.

Employment History

Operations Manager , BC AirCare Program 2012 to present Senior Project Engineer , BC AirCare Program Oct 1994 to 2011 Manager of Emission Testing and Standards (A cting), BC AirCare Program Aug 1998 to
Dec 1998 Emissions Testing Specialist, BC AirCare Program Apr 1993

- Sep 1994 Honorary Research Associate, UBC Department of Statistics Sep 2000 to present
Instructor, BCIT School of Transportation Apr 1998 to Jun 2003
Instructor, Columbia College, Burnaby, BC Jan 1993

- Apr 1993, 1994 and 1996A

- 6

Continuing Education Instructor, University College of the Fraser Valley Jan 1995

- Mar 1995 Substitute Teacher, Abbotsford School District (#34), BC Oct 19 92

- Mar 1993

Consultant to Monodraught Ltd., High Wycombe, England 1988 to 2007 Senior Lecturer, University College Salford, School of Engineering, Salford, England Jan 1986

- Aug 1992 Airframe Systems Engineer, British Aerospace, Woodford, England Aug 1978

– Aug 1982

Education

PhD. Flight Simulation , University of Manchester Oct 1983

- Dec 1985 MSc. Mechanical Engineering, University of Manchester Oct 1982

- Oct 1983 BSc.(Hons) Aeronautical Engineering , University of Bristol Oct 1975

- Jun 1978

Project Experience

Automotive Emissions Testing and Repairs

Light

- duty gasoline vehicles

SS reviewed and developed ASM and idle test standards for the BC AirCare program starting in 1995. The final version was created in 2000, and is still used for 1991 and older vehicles. These standards are a function of vehicle weight for ASM tests, and of engine size for idle tests, as well as accounting for vehicle model year. Also implemented in 2000 was reporting of average passing readings which give motorist s an indication of what is normal for their vehicle. These readings were implemented for IM240 as well as for ASM and idle.

In 1998 to 1999 SS ran two projects to test gas caps on in

- use vehicles in BC. In the first project the gas caps of vehicles undergoing regular inspection were tested in order to assess the incidence of leaking caps on in

- use vehicles. This project provided new caps for all vehicles that showed any leakage, and the old caps were retained for the second project. The second project measured mass flow leakage rates as a function of pressure differential, using

a custom

- built system.

From 2002 to 2008 SS ran a series of projects to establish the efficacy of subsidizing emission repairs that would otherwise not be performed because of repair cost limits. The projects achieved emission reductions by providing repair subsidies for light

- duty vehicles
that had failed emissio n inspections, and which would otherwise receive a Cost Waiver. Administration of the program required accurate assessment of individual potential benefits from diagnostic information, and the specification of appropriate repairs. With these controls in p lace it is a cost

- effective policy.A

- 7

Working with faculty and graduate researchers in UBC Dept of Statistics during 1999 to 2001, SS developed a knowledge

- based expert system that can assist repair mechanics in selecting the repair actions that are most l ikely to reduce a vehicle’s emissions to below allowable I/M maxima. The expert system is based on inspection and repair data collected by the British Columbia AirCare I/M program. The system is an observational model based on what was done, and appeared to work, in real reported cases. It does not use the type of rules

based approach which could be developed from a failure modes and effects analysis. Its most efficacious implementation is as part of an overall package which also includes this other
type of information. Access to is via a www interface, and allows its responses to incoming queries to be generated from the most up

- to

- date data available.

From 2003 to 2005 SS develop a method to report CO2 emission rate and fuel consumption
from IM240 tests . The difficulty is that existing IM programs do not all use a standardized test duration or test method and only a mass

- emissions test, driven over a specific cycle can be considered. The calculation of fuel consumption from the quantities recorded in a

c omplete IM240 test is a simple function of the masses of CO2, CO and HC, the carbon weight fraction of the fuel, and the distance driven. A problem arises when trying to compare
results from tests which have fast

- passed and have been terminated at differen t times, because the rate of fuel consumption is not constant through a test. So when a test terminates early the actual measured rate of fuel consumption could be much less than, or much more than the rate that would be achieved if the test had gone to f ull duration. This project developed methods to project full

- duration fuel consumption from tests which actually fast

passed.

In 2006 and 2007 SS led a multi

- stakeholder team to develop the communications strategy and formats for reporting CO2 emission r ate and fuel consumption for every vehicle tested by the BC AirCare program. The result was successfully implemented in June 2007

From 2007 onwards SS has organised and run vehicle emissions clinics at various locations in BC, YT and AB in response to req uests by local air quality organizations. These clinics introduce motorists to the need for proper emissions maintenance and repair, as well as establishing a picture of the in

- use emissions performance of vehicles in non

- IM areas.

In 2003 SS developed a method that used total exhaust carbon to identify IM240 test problems where exhaust was being lost from the sampling system. This enabled invalid tests to be identified and aborted when they reached 30 seconds into the test cycle. The method was implemente d for all IM240 tests in BC in 2004. It saved operational time and avoided many errors of omission.A

- 8

Light

- duty diesels

In 1999 SS created a new transient light

- duty diesel test that was implemented by the BC AirCare program in 2000. The test used the sec ond phase of the IM240 driving cycle and looked for peaks in exhaust opacity. It was subsequently adopted by KOTSA for all diesel vehicle inspections in South Korea as a replacement for their lug

- down test

Heavy

- duty diesels

In 1996 SS was part of the team that launched the BC on

- road heavy

- duty diesel inspection program. This was the first program anywhere to use the SAE J1667 test. This project included everything from specifying, purchasing and commissioning equipment, to policy making, establishing a network of independent repair facilities, industry communication and liaison, and regular analysis of inspection data after implementation.

In 2010 SS was a member of the industry advisory working group for Metro Vancouver’s development of regulations f or off

- road heavy

- duty diesels. These are primarily construction equipment.

Analysis of Vehicle Population and Use Data

In 1996 SS developed a method for assessing in

- use annual kilometers travelled from odometer readings recorded during annual inspecti ons, and which tracked the readings of individual vehicles. Previous estimates were limited by database considerations to comparing average values from different calendar years and vehicle model years. SS has used this
method for all su bsequent vehicle ann ual kilomet e r estimates.

Starting in 2002 and continuing into 2012 SBA has undertaken a series of analyses for Environment Canada Pollution Data Branch to define a detailed fleet profile, by GVWR, by fuel type, by model year, for all the vehicles in Canad a, separated by jurisdiction and down to the level of FSA. The projects have drawn on various available data sources, including provincial registr ation files, Drive Clean data, AirCare/ACOR data, and Polk and DesRosiers summary data. From these data source s, the vehicle fleet can be defined by province/territory, by Gross Vehicle Weight Rating (GVWR), by fuel type, by model year, by postal code, at quarterly intervals from 1980 on.

In 2000, working with GWT Consulting, SS created an on

- board system to log in

- use driving
behavior . The purpose of this study was twofold: to create a driving trace representative of typical commuter driving; and to develop better estimation techniques for cold

- transient fuel use and emissions. Study data was collected from a co mbination of rental and volunteerA

- 9

vehicles. Collected data comprised vehicle position, elevation, speed and heading
information from an on

- board Global Positioning System (GPS); and a list of vehicle parameters extracted from the vehicle data

- stream via t he On

- Board Diagnostics (OBDII) connection. This project highlighted the amount of ‘off

- cycle’ operation that is included in normal driving, and its contribution to overall emissions, with specific emphasis on cold

- start fuel consumption.

Reporting Emission Reductions

Since 1995 SS has performed biennial analyses of the emission reductions achieved by the BC AirCare program. The first evaluation in 1994 was by dKC and subsequent evaluations built on and expanded this initial work. The par ticular feature of the approach is the use of in

program data which includes very robust datasets of mass emission data that define the emission performance of vehicles that pass and fail the IM tests and those that are repaired. The approach and the bienn ial reports were heavily referenced (although anonymously because they were not a US source) by Eastern Research Group in their report to the US congressional committee on IM program effectiveness reporting in 2004.

Each year since 1999 SS has performed e missions reduction evaluations of the Light

- Duty Component of Drive Clean Vehicle Emissions Inspection and Maintenance Program. Sub

sections of this work involve data analysis covering different aspects of the Drive Clean program. There are almost 7 milli on vehicles registered in Ontario, and each year approximately 3 million inspections are performed. We have analysed and modeled failure rates, effects of program policies, fleet characteristics, repair effectiveness, and program
benefits. The emission red uction estimates use mass emission factors derived from BC
datasets.

In conjunction with GWT Consulting since 2001, SS has undertaken analyses of Dive Clean heavy

- duty diesel test data for input to the malperformance model. The vehicle counts and
SAE J166 7 opacity reading frequency distributions define the probable occurrence of different types of malperformance, and these are then related to PM and other emission rates, with the overall result being an evaluation of the emission reductions attributable to the heavy

- duty inspection program.

In 2005 SBA did an evaluation of the suitability of NOx reductions from motor vehicle emission reduction projects for use under Ontario Regulation 397/01 which is an emissions offset regulation. The overall context was Ontario regulation 397/01 which governs annual limits for nitric oxide and sulphur dioxide e missions from the electricity generating sector. To satisfy the requirements of these limits a power producer must submit Emission Reduction Credits (ERC) which matches actual annual emissions tonne for tonne. The ERCsA

- 10

are derived from reduction projects u ndertaken by non

- regulated facilities and proponents. The purpose of this project was to review the potential for NOx emission reductions from transportation sources through programs that address human behaviour.

In 2010 SS led a project for Retire

- Your

- R ide (the Canadian national vehicle retirement program) to test the emissions of a 150 scrapped vehicles and compare with the emissions from a range of new vehicles that were typical of those being chosen as replacements. This project included PM measuremen ts as well as HC, CO, NOx and CO2, and confirmed that the claims made for the benefits of vehicle retirement as an emissions reduction approach
were more than reasonable.

Other Vehicle Emission Reduction Projects

SS did significant work with the BC alter native fuel conversion industry throughout the 1990s, after uncovering the major problems conversions were experiencing in the inspection program. This included writing the BC Alternative Fuel Conversion Policy; a series of projects to establish the causes of poor emissions performance and how it could be remedied; and ongoing comparison testing for various public fleets. This led to some innovative work in conjunction with the Workers Compensation Board of BC and with UBC Environmental
Health on remediatin g problems with indoor equipment such as forklifts and ice

- resurfacers.

For the Vancouver Airport Authority, in 2005 SS created a simple method to determine which taxis qualified for licensing incentives based on their environmental performance.

Selected publications

Society of Automotive Engineers Technical Papers

1. "British Columbia Vehicle Emissions Inspection and Maintenance Program Experience of Alternative Fuel Vehicle Conversions" S.J.Stewart, D.I.Gourley, S. Loo SAE 941913

2. "The Certification a nd Monitoring of Technicians and Repair Centres in British Columbia's Aircare Program" S. Loo, S.J.Stewart, D.I.Gourley, SAE 950483

3. "An Evaluation of the Effectiveness of Repairs in British Columbia's AirCare Program" S.J.Stewart, S.Loo , D.I.Gourley SAE 950482

4. "Correcting Emissions Problems in Existing Propane and Natural Gas Vehicles in British Columbia" S.J.Stewart, D.I.Gourley, S. Loo SAE 952380

5. "Repair Effectiveness Indices for the British Columbia Vehicle Emissions and Maintenance Program" S.J.Stewart, S.Loo SAE 961700A

- 11

6. "Study of In

- Use Alternative Fuel Vehicle Emission Performance under EPA and BC AirCare Test Cycles" A.Inglis, C.Prakash, S.J.Stewart SAE 961709

7. "The Development of Advanced Technician Training to Meet the Demands of Enhanc ed Vehicle Emissions Inspection and Maintenance Program Implementation' D.Horrobin, R.MacGregor, T.Wood, R.Plett, J.Marchant, S.Loo, S.J.Stewart SAE 961701

8. "A Study of Mileage Accumulation Rates of Light

- Duty vehicles in the Lower Fraser Valley" S.J.St ewart SAE 961702

9. “Quantification of Evaporative Emissions from Defective Fuel Filler Caps” S.J.Stewart, J.Wong, L.Jang, C.Hui, D.Meggy. SAE 2000

- 01

- 1171

10. “Cold Start Impact on Vehicle Energy Use” G.W.R.Taylor, S.J. Stewart SAE 2001

- 01

- 0221
“Emission s Performance of In

- Use Alternative Fuel Vehicles” J.Wong, D.Gourley, S.J.Stewart, SAE 2001

- 01

- 367812 

### B. Appendix B: Charge Questions

Particulars Page

1. Letter to Panel Participants with Charge questions B 1

- B 4B

- 1

TO: Robert F. Klausmeier (de la Torre Klausmeier Consulting, Inc.)

Stephen A. Leydon (SEEN/Georgia DNR) Stephen J. Stewart (Pacific Vehicle Testing Technologies)

FROM: Dileep K. Birur

CC: Michael P. Gallaher, Tony Lentz

DATE: February 19, 2012

.

SUBJECT: Charge Questions for Peer Review of ERG’s study on “Evaluation of the

Effectiveness of On

- Board Diagnostic (OBD ) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles.”

The U.S. EPA’s Office of Transportation and Air Quality is currently reviewing Eastern Research Group’s (ERG’s) draft report on analyzing high evaporative (evap) emission field data collected in the Denver area during the past 4 years. The information gathered in the ERG’s “high evap” studies and the pretesting studies tha t preceded them indicated that Inspection and Maintenance (I/M) OBD results may have the potential to identify high evap emission vehicles through one or more measures of evap emissions. In the ERG’s report, EPA is further investigating the possibility tha t relationships may exist between several different measures of light

- duty vehicle/light

- duty truck (LDV/LDT) evap emissions and the variables that influence these emissions. If these relationships exist for evap variables, then they can be used in a
model ing context to expand EPA’s MOVES predictive capacity for LDV/LDT evap emissions.

In support EPA’s Tier 3 rulemaking, EPA is seeking the reviewers’ expert opinion on its contractor’s (ERG) report of evaporative emission field data and related OBD result s for light

duty passenger cars and trucks. Thank you for agreeing to review this report. We are submitting this document to you for a peer review of the methodology, and the validity of the data and assumptions that go into it. EPA has provided direction and charge questions for this review and these are included below. If it is necessary, RTI will arrange a teleconference call so that EPA can respond to questions from individual reviewers on the material that was provided for review.

The review will inv olve a written report that includes the response to the charge questions and any additional comments you may have, e.g., margin notes on review materials. Comments should be provided in an enclosure to a cover letter that clearly states the reviewer’s name , the name and address of their organization if applicable, which model review documents/media wereRobert F. Klausmeier

Stephen A. Leydon Stephen J. Stewart February 19, 2012.

B

- 2 received by the reviewer and which were actually reviewed and a statement of any real or perceived conflict(s) of interest. Please submit the completed revi ew reports and the signed invoice (invoice forms will be sent separately by RTI’s contract office) to RTI by March 05,

2012. The ERG’s report to be reviewed is attached with this letter. Please keep your comments
confidential until the initial release of the peer review report by the EPA. If you review the document as a team, please provide the details of your team members as well

.

Elements to be addressed in the Charge to the Reviewers of ERG’s report:
“Evaluation of the Effectiveness of On

- Board Diagnostic (OBD) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles.”

The focus of this report is on the analysis of evaporative emissions and related data collected on light

- duty vehicles equipped with OBDII systems, which are present on 1 996 and newer model year vehicles. In each of the three high

- evap studies (Lipan, Caryl, and Denver, 2010), many vehicles received measurements relevant to this report. In these three studies, a small fraction of these vehicles were successfully recruited as participants, and the remaining
vehicles were non

- participants

— the combination of study/participant status producing the three separate datasets. The analysis looks to identify any relationships which may link the several different measures of evaporati ve emissions and the variables that influence evaporative emissions. If they exist, the report summarizes the evidence for using them in a modeling context which may be useful to the development of EPA’s MOVES model. Correlating tailpipe emission measureme nts using IM240 tailpipe measurements and the OBD system has always been challenging and has been well documented. In this report, the relationships which are explored are between the results of high evaporative vehicle emission
field studies and:

 Onboard diagnostic trouble codes (OBD DTCs); indicate vehicle parameters/conditions, including whether vehicle has seen high exhaust or high evaporative emission levels. During routine IM station inspections, OBD inspections are performed on 1996+ vehicles. OBD co des are discrete variables, with values of
‘set’ or ‘not set’.

 Portable or laboratory

- based SHED results; an assessment of evaporative emissions by testing each high

- evap vehicle in a SHED (evap emission results are continuous variables with units of grams HC/15minutes).

 Modified California Method (MCM) inspection; an under

- hood and under

- body olfactory, visual, and electronic HC

- sniffer check, for each high

- evap vehicle. MCM variables are generally discrete and the inspections provide ‘smell/no

- smell’ (not edRobert F. Klausmeier

Stephen A. Leydon Stephen J. Stewart February 19, 2012.

B

- 3

by the inspector), apparent condition of various fuel system and evap control system components and ‘detect/no

- detect’ by the electronic HC sniffer. Participant vehicles and non

- participant vehicles received different sets of emission tests
in each of t he three studies covered in this report. Non

- participants (I/M

- compliant vehicles only) received RSD measurements and OBD inspections, which reported back diagnostic trouble codes (DTCs) for 1996 and newer model year vehicles. Participants (I/M volunteers for high evap emission vehicle testing) received, in addition to RSD and OBD measurements, a measurement of evaporative emissions by placing the vehicle in a SHED. In the Lipan and Ken Caryl studies, a portable PSHED was used. In the Denver 2010 study, a p ortable SHED was used for the vehicles tested at CDPHE’s West Tech Center and a laboratory SHED was used for vehicles tested at CDPHE’s Aurora test facility. Participants also received MCM inspections. The analysis described below is broken out by:

 Model Y ear;

 OBD Evap readiness monitor status; and

 Presence of enhanced evap system for 1996

- 1998 vehicles. (break

- outs were made to the finest level of detail possible; however, this was dependent on the sample size for each category)
The three different dataset s were used in this study to determine if high evaporative emitters, defined by the portable SHED, were appropriately identified by the OBD system on the vehicle. The results for this analysis seem to indicate that many vehicles with high portable
SHED val ues probably do not have evaporative DTCs set. For vehicles with high values from portable SHED testing, ERG attempted to make an assessment of the source of the vapor leak. These assessments suggest that about half or more of the high emitters identified should have been identified via the OBD system but were not. The preliminary analysis suggests that OBD systems were unable to identify 50

– 70% of the potentially high evaporative emitters in these study groups. The lower end of the range, 50%, is based on the known occurrences of when the OBD system should have detected the leak from a known source as shown in “technician” comments. The higher end of the range, 70%, includes the unknown leak sources which are likely to be located in the vapor space of the f uel system, which is too tightly packed for the HC sniffer to reach, as opposed to the fuel line connection points or the fuel rail, which are liquid leaks and not detectable by the OBD system. Additionally, the OBD results indicated that vehicles with hig h portable SHED values were likely to have evap DTCs set. However, it was seen that high portable SHED values also indicated the likelihood of having an exhaust HC DTC set. Vehicle age was found to be insignificant statistically for identifying vehicles wi th evap codes set, but was significant for identifying vehicles with exhaust codes set. In their comments, reviewers should distinguish between recommendations for clearly defined improvements that can be readily made based on data or literature reasonably availableRobert F. Klausmeier

Stephen A. Leydon Stephen J. Stewart February 19, 2012.

B

- 4 to EPA and those improvements which are more exploratory or dependent on information not readily available to EPA. Any comment should be sufficiently clear and detailed to allow a thorough understanding by EPA or other parties familiar with the report. EPA requests that the reviewers not release the peer review materials or their comments to anyone else until the Agency makes its report and any supporting documentation public. If a reviewer has questions about what is required in order to complet e this review or needs additional background material, please direct the reviewer to contact RTI’s project manager for this effort. If a reviewer has a question about the EPA peer review process itself, please have the reviewer contact Ms. Ruth Schenk in E PA’s Quality Office, National Vehicle and Fuel Emissions Laboratory by phone (734

- 214

- 4017) or through e

- mail at schenk.ruth@epa.gov

.

### C. Appendix C : Questions and Answers Provided During

the Review Process

Particulars Page

1. Questions and answers provided during the review process C 1C

- 1

Question from Rob Klausmeier to RTI (email dated February 23, 2012):

The following sentences appear to be inconsistent. 2nd paragraph page 3, 2. The results for this analysis seem to indicate that many vehicles with high portable SHED values probably do not have evaporative DTCs set. 3rd paragraph page 3, 2. Additionally, the OBD results indicated that vehicles with high portable SHED values were likely to have evap DTCs set. Could you cla rify which statement is correct?

EPA response to RTI to question from Rob Klausmeier (email dated February 27, 2012) :

T he first paragraph below (with the 50

- 70%) is the correct charge question. The second o ne doesn't make much sense, as you will see when you read the revised report which was sent.

### D. Appendix D : Cover Letters

Particulars Page

1. Cover letter from Mr. Rob Klausmeier D 1

2

. Cover letter from Mr. Stephen A. Leydon D 2

3

. Cover letter from Dr. Stephen J. Stewart D 3D

- 1D

- 2D

- 3

### E. Appendix E : Review Reports

Particulars Page

1. Review Report from Mr. Rob Klausmeier E 1

- E 2

2. Review Report from Mr. Stephen A. Leydon E 3

- E 6

3

. Review Report from Dr. Stephen J. Stewart E 7

- E 11E

- 1 

### Review

- 1 by: Mr. Rob Klausmeier

.

To: Dileep K. Birur, RTI Int e rnational, I n c.

Fro m : Rob Klau s m eier, de la T orre Klaus m eier Consulting, Inc.

Subject: Comments on the Report “Evaluation of the Effectiveness of On

- Board

Diagnostics (OBD) S y stems in Identif y ing Fuel Vapor Losses from Light

- Du t y Vehicles”

As requested by RTI, I have reviewed E R G’s report evaluating the effectiveness o n

- board diag n ostics (OBD) syste m s in identifying vehicles with significant fuel vapor losses. I have
been involved in m obile s ource evaporative e m issions m odeling since 1985 and have helped

m any states i mpl e m ent OBD inspectio n s as part of th e i r inspe c tion/ m aintenance (I/ M )

pro gr a m s. As a result, I reco g nize t h e i m porta n t consequences of the initial findings of this
report

– that OBD syste m s m a y not identify a large fraction of the fleet (50

- 70%) with

e x cessive evaporative e m issions. This finding affects both state i mple m enta tion plans with

r eg ards to the eff

ecti v eness of I/M progra m s and future strategies for phasing out Sta g e II
syste m s and instead relying on on

- board refueling vapor recovery (ORVR) syste m s to control refueling e m issions. I believe t h at the data collected an d analy z ed in this study are inadequate to make these conclusions, and that the report should not be released until sampling concerns listed below are addressed. My comments are broken into m ajor and
editorial comments.

Major Comments:

I have several concerns over the sample s i ze and m ethodology used to procure vehicles for this test program and accor d ingly the re p rese n tativeness of the test sa m ple. In addition, I
have concerns over so m e of t h e OBD assu m ptions. Below are my com m ents:

I recog n ize the difficulty in identifying vehicles with high evaporative e m issions, but this reality does not negate the fact that you cannot m ake a definitive conclu s i on o n OBD perfor m ance based on 10 hi g h e m itting vehicles.

The report should present m ore details on how vehicles were procured for the PSHED test. The report notes that priority was given to recruiting vehicles that had high evaporative indices as deter m ined by RSD m easure m ents. The report should provide
details on the nu m ber of vehicles screened by RSD, the fraction of
the fleet that had high evaporative indices based on RSD, and the ulti m ate fraction of
the fleet that received PSHED and O B D tests. Info r m ation on the year, m ake and

m odel of vehicles that rece i ved PSHED tests would be useful.

The repor t appears to m ake the assumpt i on that vehicles id e nti f ied f or t e sting because they had high evaporative indices a r e representative of all vehicles that have highE

- 2

evaporative e m issions. The report does not provide evidence that vehicles with high evaporative indices b a sed on RSD are necessarily typical of all vehicles with high evaporative e m issions. For exa mple, vehicles with vapor leaks that do n o t create a hydrocarbon pl u m e while t h e vehicle is driven b u t still create significant vapor losses
du ring hot

- soak and diurnal conditions would not have high RSD evaporative indices.
As a resul t , I have concerns over the conclusion that OBD misses 5 0

- 7 0 % of the hi g h

evapor a ti v e e m itters. I nstead, the r eport c an only concl u de OBD m i sses 50

- 70% of the high e vaporative e m itters as deter m ined by the RSD evapor a tive index, with the sa mple
size caveat.

The authors should collect and analyze data from I/M progra m s, California roadside
emission tests, and other sources to validate, if possible, the assumption that vehicles with high evaporative e m issions identified in this study are typical of all vehicles with high evaporative e m issions.

The report assu m es that because the evapo r ative monitor is r e ady the OBD system has recently checked t h e evaporati v e e m is sion control syste m . After the evapor a tive
monitor be c o m es ready it stays ready until codes are c l eared. Because most vehicles, particularly older m od e ls, have stringent criteria to run the evaporative monitor, a vehicle could likely have a problem with th e evaporative e m ission control syste m , be ready, and have the MIL

- off (with no DTCs). The OBD system m ay eventually ide n tify

t h e problem, set an evaporative e m ission DTC and turn the MIL on.

The report notes that so m e vehicles with high evaporative e m is sions had exhaust related DTCs but no evaporative DTCs. Did ERG investi g ate the ena b li n g criteria for
the eva p orative DTCs to see if t h e p r esence of exhaust DTCs m i ght have kept the evaporative m onitor from running?

The report should attempt to co m pare the distribution of dia g nostic trouble codes (DTCs) observed in this sa mple with t h e distribution of DTCs found in vehicles tested in I/M progra m s. In Connecticut, for example, 17% of the vehicles with illu m i nated
MILs have e vaporative e m ission DTCs. The t op evapor a tive DTCs in Connecticut (in descending ord e r) are P0442, P0455, P0440, P0441, P0446, and P0456. The four evaporative DTCs in ERG’s study are P0442, P0443, P0451, and P0457. Only one appears on Connecti c ut’s top evaporative DTC list.

Also, even though it’s not the m i ssion of the report to evaluate the RSD evaporative index, the fact that only 29 out of 157 vehicles exceeded t h e 0.3g/Qhr threshold raises
concerns about the e ffectiveness of the RSD based index in identifying vehicles wit h

e x cessi v e evaporative e m issions.

Editorial C omments:

I have provided a m arked

- up copy of the r e port that contains editorial comments.E

- 3 

### Review

- 2 by: Mr. Stephen A. Leydon

TO: Dileep K. Birur (RTI International)

FROM: Stephen A. Leydon

DATE: March 18, 2012

SUBJECT: Review of “Evaluation of the Effectiveness of On

- Board Diagnostic (OBD) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles” V ersion 9 dated February 17, 2012

This is an evaluation of Eastern Research Group’s (ERG’s) final report on evaluating the effectiveness of Second Generation On

- Board Diagnostic (OBD) systems through a comparative analysis of evaporative emissions field data collected in the Denver area.

Several studies were performed through Cooperative Research and Development Agreements between the Assessment and Standards Division (ASD) within the Office of Transportation and Air Quality (OTAQ) of the United States Environmental Protection Agency (EP A) and the Colorado Department of Public Health and Environment (CDPHE), with support provided by ERG. Data was collected at the Denver, Lipan, and Ken Caryl Inspection and Maintenance (I/M) lanes using a Portable Sealed Housing for Evaporative Determinati on (PSHED) and remote sensing devices. Because the studies took place at existing I/M lanes, OBD information was also available for each vehicle. The efforts span a four

- year span covering 2008

– 2011 at three locations in and around Denver, Colorado as noted above. Since the CDPHE operates IM240 lanes capable of capturing exhaust gas readings and perform OBD II tests on 1996 and newer vehicles it was beneficial to also gather evaporative emissions on participating vehicles as well. These locations were also able to perform “SHED” testing which provided a unique opportunity to perform and gather data for this study. Since OBD II vehicles are equipped with constant

- monitors and enhanced fuel metering systems the tailpipe emissions are extremely low by de sign. However, “vehicle emissions” don’t come exclusively from the tailpipe of the vehicle but from a variety of sources. OBD II equipped vehicles are designed to monitor some potential non

- tailpipe emission points such as fuel vapor leaks from the fuel storage system. When an OBD vehicle senses a possible leak in the vapor system the malfunction Indicator Light (MIL) is required to illuminate to warn the driver of the problem. In response to the MIL illuminating, the driver is supposed to effect repair s to correct the problem causing the elevated emissions. For the purpose of these studies it is fortunate that not all motorist pay attention to an illuminated MIL. If the illuminated MIL were heeded as expected by EPA, then none of the vehicles in the f irst two studies would have arrived with active DTCs. The responsive motorist with an illuminated MIL however may be the reason so few participating vehicles had any DTCs.E

- 4

In the analysis section the assumptions regarding the OBD tests in the first two studies that were taken shortly before the SHED test or the previous day are valid in my opinion since very few vehicles change their emissions status over night. In the first two studies the OBD was evaluated up to a day prior to the SHED testing. With OBD, however the MIL can illuminate on the next key cycle after an OBD evaluation has been performed but that does not mean that the emissions just exceeded the limits. In many cases the emissions have been elevated for up to several key cycles prior to t he illumination of the MIL. No mention was made that any participating vehicle had the MIL off during the OBD II evaluation but illuminate after the OBD test during the SHED test(s).

The “Denver 2010” repair effectiveness study indicates 68 vehicles were recruited using RSD measurements around Denver. This document does not specifically state that the vehicles recruited by mail had very high RSD readings and were considered “gross polluters” or what threshold was the determining factor for selection but it does not make sense that “clean vehicles” would be solicited for the repair study. This group had a 100% OBD/SHED match rate presumably because the vehicles were recruited and all tests were performed on location during one visit, which is the ideal sit uation. The establishment of the PSHED at the Denver facility and correlating the readings to LSHED ensured that, for all practical purposes the readings are to be considered the same and was also a great benefit for the continuity of the study.

Revie w of the timeline of the several studies and the number of “dropped” tests indicates that as each study was completed and analyzed that lessons were learned and improvements were made before beginning the next study. These adjustments allowed all tests in the Denver 2010 study to be paired. This important due to the costs of performing the study of each vehicle in a study in terms of man

- hours, expenses, and other costs not producing an end product. The summary in table 3

- 1 illustrates the point very wel l since the first set of data had a 29% dropout rate. This was cut in half in the following study to 14.6%, followed by a 0% dropout rate in the last (Denver) study.

Table 3

- 2 is an interesting study in the further filtering of questionable data. The pe rcent of participating vehicles with the specific evaporative monitor ready was 95% (149/157). This high percentage of a “ready” is interesting to me given the mix of older vehicles in the study. It is widely accepted that the 1996 through 1998 model yea r OBD II systems did not easily get the evaporative monitor to a ready state. As a result the EPA allows up to two monitors to be not ready in the 1996

– 2000 model years. In fact vehicle HE

- 6725 had a faulty fuel cap that was not detected even though the monitor indicated it was ready.

Reviewing the two tables 3

- 3 and 3

- 4 the data indicates that if a vehicle is going to exceed the 0.3g/Qhr limits then it is very likely to exceed the 1.0g/Qhr as well. Out of 27 vehicles that exceeded the 0.3g/Qhr 20 al so were in excess of the 1.0g/Qhr limit which is 74%.
Review of the data in these two tables also indicates that 3 out of the 4 vehicles with an evaporative DTC were also over the 1.0g/Qhr, which is 75%. The data strongly suggests that if aE

- 5

vehicle is ove r the 0.3g/Qhr then it will more than likely be over 1.0g/Qhr as well. The data in these tables also suggest that if an evaporative DTC is set (MIL on) then the vehicle will definitely be exceeding the 0.3g/Qhr threshold and that it more than likely excee ds the 1.0g/Qhr as well. This finding indicates that when the OBD II system evaporative monitor is ready and has run and finds a problem (sets a code) there are definitely high measurable SHED emissions that can also be detected by RSD. This is a positive finding for programs that continue to perform RSD for program evaluation. It would lend credibility to RSD findings of “high emitters” regardless of the results of paired OBD records. This finding provides the reason why OBD II equipped vehicles that ar e flagged as gross polluters by RSD will pass an OBD II inspection. This is something we have seen in Georgia in past years.

Unfortunately the findings also provide a strong indication there may be many vehicles with high emissions where the OBD II syste m has not identified an issue that it is supposed to be monitoring. Some factors to consider here are that the evaporative monitors have a lot of enabling criteria that must be met to allow it to “run”. Ambient temperature greatly affects this monitor, a s does the amount of fuel in the fuel tank. Given the current state of the economy many motorists only purchase small amounts of gasoline at each refueling due to the cost. This aggravates the evaporative monitor’s ability in particular to become ready s ince the fuel level typically must be at least a quarter of a tank or more to enable the evaporative monitor to become ready and run. Another reason the errant emissions identified by RSD and measured by SHED testing are not identified by OBD II is that s ome emissions are due to leaks in systems not monitored or by liquid leaks such as oil, fuel, and high VOC content washer fluid (Atlanta Georgia has such a restriction).

In the evaluation report there is mention that 1981 and newer vehicles were asked to participate in at least some of the studies (Lipan, Caryl). OBD II vehicles began with 1996 model year. This difference in model years could be the reason why so many ve hicles had no OBD readings to pair with SHED data, a loss of 29% and 14% in Lipan and Caryl studies. No specifics are given provided for the high drop out due to no OBD II readings, but this would certainly account for the high number. The RSD identified vehicles in the Denver study had a 0% drop out due to non

- pairing presumably due to only 1996 and newer vehicles being sent
recruitment letters.

Reviewing comments by inspectors performing the MCM two specific vehicles. Both

LIP

- 254 and CRL

- 568 have th e comment “Gas cap IM failure.” In addition, HE

- 6725 has the comment “Gas cap was cause of leak.” These comments infer that a fuel cap pressure test was used to determine the gas cap leak failure. No mention in the study is made with regard to the

use o f a separate fuel cap integrity test however 1975 and newer vehicles have this check done in Colorado according to information found on the OBD Clearinghouse website. If this is the case then only three vehicles had a leaking gas cap and one gas cap was m issing (HE

- 6702) out of 180 vehicles. The OBD systems in these four early OBD II vehicles did not store an evaporative DTC because they do not have enhanced evaporative monitoring. This finding may result inE

- 6

non

- enhanced evaporative monitored vehicles be ing given more weight as polluters in modeling due to their demonstrated lack of being able to identify high evaporative emissions or even the presence of a fuel cap at all.

Observing the results of the High and Low SHED Pairs data in table 3

- 6 it become s apparent that newer vehicles (2004 and newer) have a definite ability to control evaporative emissions better than earlier years. This may be in part to lower mileage and partly due to manufacturers getting better at sealing fuel fittings. Older enhanc ed evaporative OBD II vehicles (2000

- 2003) show a fairly low number of high value pairs (10%

- 14% over .3g/Qhr) while, as expected the oldest OBD equipped vehicles have the highest SHED numbers (50% over .3g/Qhr). This data also indicates a high number ov er 1.0g/Qhr for the older OBD II vehicles with limited evaporative monitoring. This table illustrates the older OBD II vehicles certainly are the ones most likely to be the vehicles identified by RSD and as a percentage of their model year have a large a mount of evaporative leaks undetected. As a group, the older OBD II vehicles contribute more evaporative emissions than newer and should be given a higher weight when calculating emissions from the fleet. Not all OBD vehicles were created with the

same a bilities to monitor the various systems, which is an understatement, but the reality is that there are tens of thousands of early OBD II vehicles without enhanced evaporative strategies still operating on the highways. These vehicles are contributing to h igher levels of ambient emissions wherever they travel.

These studies were performed in an area that has had I/M testing for many years, as a result, the motoring public is well aware that vehicle maintenance is important in order to pass an
emissions tes t. Motorist, therefore are more likely to maintain their vehicles especially when out of warranty. Since motorist in the study area are very likely to maintain their vehicles expectations that if this study were duplicated in a non

- I/M venue the results would show much higher percentage of >1g/Qhr and >0.3g/Qhr vehicles than were identified here. This study provides valuable data regarding the ability of OBD II to identify, or not, evaporative issues with vehicles. What is very positive is that if the M IL is illuminated with an evaporative DTC it is broken and does have high emissions.

This study also indicates that many OBD II equipped vehicles are emitting at levels identifiable by RSD and verifiable using SHED testing. For whatever reasons there a re high emitting OBD II vehicles the facts indicate there may be significant emissions coming from vehicles assumed to be very clean by federal and state agencies using statistical modeling. If this study is a good indication of real life experience, and it should be, modeling of OBD II vehicles without enhanced evaporative strategies should be weighted more than newer OBD II vehicles with enhanced evaporative monitoring. There appears a strong correlation between vehicle age
and emissions and between non

- enhanced evaporative strategies and newer OBD II equipped
vehicles.E

- 7 

### Review

- 3 by: Dr. Stephen J. Stewart

Evaluation of the Effectiveness of On

- Board Diagnostic (OBD) Systems in Identifying Fuel Vapor Losses from Light

- Duty Vehicles FINAL REPORT Version 9 Contract No. EP

- C

- 06

- 0

- 80 Work Assignment 5

- 10 Prepared by: Eastern Research Group February 17 th 2012

.
Comments by S.J. Stewart March 5 th 2012

.

Issue

The issue of interest here is whether it is possible to use real

- world OBD data regarding the

evapor ative emissions performance of in

- use vehicles as an input to inventory modeling. One commonly available source of data is the OBD data from I/M programs. But can this data provide a direct indication of the evaporative performance of in

- use vehicles? For this study the measure of actual evaporative performance was SHED (or PSHED) testing. Its results are on a continuous scale; however in this study they were divided into just ‘high’ and ‘low’, but with
two different cut

- points between the designations. The OBD results are simpler in that they are ‘yes/no’ data; either there were evaporative trouble codes, or there were not. So the question comes down to the correlation between ‘high/low’ on the SHED results and ‘yes/no’ on the presence of evaporative troubl e codes.

Pre

- Screening

The first thing to note is that participant vehicles were screened using RSD, so that only those likely to have high evaporative emissions participated. So the study tells us nothing about the prevalence of OBD evaporative trouble c odes in vehicles that were deemed un likely to have high evaporative emissions. The data do show that all participants with evaporative DTCs did have high SHED results, but that is simply to be expected if the pre

- screen was effective. This leaves wide open the question of whether an OBD evaporative DTC is a good indicator of high evaporative emissions.

Because DTCs are not cleared until 40 warm

- up cycles have occurred without re

- occurrence of the problem indicated by the DTC, there is clearly a possibility that the presence of an evaporative DTC does not indicate a current evaporative problem, but only that one did exist at some recent time. For this study the pre

- screen sought to ensure that only high evaporative vehicles were likely to participate, and on ly ‘hard’ DTCs (with MIL on) were considered. Future investigations might also include vehicles that would have been screened out from this study, and also those with ‘pending’ evaporative DTCs.

The RSD pre

- screen does not appear to have been very effect ive. Out of 149 cases shown in Tables 3.3 and 3.4, only 27 had evaporative emissions over 0.3g/Qhr, and only 20 were overE

- 8

1.0g/Qhr. So only about one in six, or one in seven were correctly screened by the RSD

- based index. The benefit of this ineffective p re

- screen is that the sample did in fact include a large number of vehicles that did not have high evaporative emissions, and it is nice to see that they also had (almost) no evaporative DTCs.

SHED vs. OBD

SHED results indicate the total rate of evaporativ e emissions from a vehicle but tell us nothing to help pinpoint where on the vehicle is the source of those emissions. This is in complete contrast to OBD systems which test only specific aspects of the vehicle hardware and control system, and thus comple tely ignore all the other possible sources of evaporative emissions.

There are only two types of evaporative problem that OBD does detect: basic evaporative OBD only checks the canister purge solenoid and flow rate; and enhanced evaporative OBD also looks for vapour leaks equivalent to a certain size of hole. So high SHED r esults caused by liquid leaks or tailpipe residuals could never be detected by current OBD evaporative monitors.

Modified California Method

The study used the Modified California Method in an attempt to identify what might be the causes of poor correlatio n between SHED

- high/low and OBD

- yes/no. The MCM inspection tried to identify the source of the emissions, and also made a subjective assessment of whether the OBD system should have been able to detect the problem. The assessment is based on a non

intrusiv e investigation, which often did not pinpoint the source of evaporative emissions, so it is not certain that only vapour leaks were included

– some cases may have been small liquid leaks into inaccessible locations, and from the comments it appears that in some cases the source could have been residual vapours from the exhaust system. So the deductions from the MCM tests are also somewhat speculative as well as being subjective. In guessing whether the leak should have been detected by OBD, they also assume that the OBD system would have had the chance to detect the problem, but this may not have been the case. It would be valuable to take the investigation further using an evaporative testing cart to measure vapour leaks rates.

Readiness

The issue of readi ness is described in the report, and the focus of the results is on cases where the evaporative monitor was reported ‘ready’. Experience with OBD data from the British Columbia I/M program indicates that the evaporative monitor is the one that is most unli kely to be ‘ready’ when a vehicle is presented for inspection 1 , and it can also be the most difficult to reset after
codes are cleared 2 . However, I/M programs have ‘readiness criteria’ which allow for one or more monitors to be ‘not ready’ at the time of inspection, and therefore I/M programs do not do

1 In 2011 in BC there were 252,497 vehicles that received OBD inspections, from model years 1998 to 2004. The number where the evaporative monitor was ‘not ready’ was 46,650, or 17.7% of the total. The rates of ‘not ready’ for the other non

continuous monit ors were CAT 9.4%; OXY 5.3%; HOXY 2.8%; EGR 6.0%
2 When vehicles returned for re

- inspection the EVAP monitor was not ready in 7,591 cases out of 14,537, which is 52.2%.E

- 9

much to ensure that evaporative monitors ever do actually run. For those vehicles where the evaporative monitor has not run, the OBD data is obviously unable to offer any clue as to the vehicles likely evaporative emissions.

Another aspect of readiness is pertinent to this study, in that once a monitor is set to ‘ready’ it will stay that wa y until the system is cleared using a scan

- tool or power is removed. The monitor will only run when the right conditions are met, and this might not be very often. It could be that the evaporative monitor ran at some considerable time in the past and did n ot detect any problems; so it set its own status to ‘ready’ and stored no trouble codes. In the time since the monitor’s readiness was recorded, a leak may have developed, but unless the enabling criteria were met, the monitor would not have run, and thus would not have even tried to detect the problem. In this case it is not valid to say that the OBD system should have detected the problem.

Results

Considering Tables 3.3 and 3.4; the four vehicles with evaporative DTCs did all have SHED results over 0.3g/Q hr, and three of them were over 1.0g/Qhr. So none of them were below 0.3g/Qhr. However, from the 149 participants, 129 vehicles were below 1.0g/Qhr and 122 were below 0.3g/Qhr, even though the pre

- screen criteria sought to only include vehicles likely to
h ave high evaporative emissions. This does not really support belief in the ability of the pre

screening system, using RSD, to identify probable high evaporative emitters.

One striking thing about the results presented in the report is how few vehicles mad e it from Tables 3.3 and 3.4 into Table 3.5. after eliminating cases where the evaporative monitor was ‘not ready’; cases of 1996

- 1998 vehicles that only had basic evaporative monitoring; and liquid leakers, only ten cases remained from the twenty shown i n Table 3.3; and only three of these had evaporative DTCs. This is hardly an adequate sample from which to draw compelling conclusions. Those three that had evaporative DTCs did have significant evaporative emissions,
and the four vehicles with the lowest SHED results did not have evaporative DTCs, so there is some suggestion that the OBD evaporative monitor might not be sensitive enough to precisely identify when a leak might cause SHED results to exceed 1.0g/Qhr, and it could be that it would only detect bigger leaks. There are also the potential problems already described pertaining to when the monitor last ran, and these may also explain something about the three vehicles in the middle, where there were no evaporative DTCs but yet the SHED results were over 6 g/Qhr. It is also pertinent that two cases without evaporative DTCs did have other DTCs, and their presence may have blocked the evaporative monitor from running. This would definitely have been the case with the Toyota Camry because one of the ev aporative monitor’s enabling criteria is for the MIL to not be commanded ‘on’. For the 1997 Mercury Villager, Alldata does not list any possible evaporative DTCs, which casts some doubt on whether this was in fact an enhanced evaporative OBD vehicle.

Conc lusionsE

- 10

The conclusions of the report are substantially supported by the data presented, but they do come across as being a little dismissive of the usefulness or reliability of OBD data as an input for modeling the evaporative emissions of in

- use vehicles . It is clear that they should only be one of a number of inputs. In truth OBD only tests a limited number of specific things, and only when other specific conditions are met.

The MCM inspectors’ opinions regarding whether or not OBD should have detected a fault have very little value because they neither hunted down the exact source of leaks nor knew when the evaporative monitors had last run.

The sample that made it through to Table 3.5, does suggest that evaporative DTCs are an indicator of high evapor ative emissions, but the number of cases that did not make it into Table 3.5 demonstrates that there are also other, very significant causes of high in

- use evaporative
emissions.

It would be tempting to conclude that when evaporative DTCs are present the y do always indicate high evaporative emissions. Although this seems probable, the pre

- screening criteria for participation in the study should have precluded that conclusion. But the pre

- screening was
ineffective and therefore does allow the conclusion th at evaporative DTCs are very unlikely to be present if SHED results are low.

The SHED results clearly indicate that the pre

- screening from RSD results is not a reliable way to identify vehicles that have high evaporative emissions. The efficacy of the pre

- screen was not the point of the study, and while adding additional process, it did not add anything to the value of
the results.

The small final data set of cases shown in Table 3.5 is really too small. Further work is necessary to enlarge this data set. The value of the data would be enhanced by improving on the MCM in order to be more precise regarding the source of emissions, and by removing the confusion of an ineffective pre

- screen.

Other Points

There are some typographic, grammatical and other err ors in the report.

Page 5, 1 st para, last sentence: Needs to mention that P2 codes are also generic and some P3s.

Page 6, 2 nd para, 10 th line. The word ‘likely’ is used as an adverb, which it isn’t. It needs to be replaced by ‘probably’. This error also occurs twice in the sentence that starts on the 14 th line of page 17; once in the 6 th line of the 1 st paragraph on page 18; and at the end of the 3 rd to last line of page 19.

Page 6, start of section 2.3:E

- 11 Second bullet: Needs to include P2 and P3 codes, and needs to indicate that P2 codes are generic, as are some P3.

Third bullet: Needs to include P#0, P#9, P#A, P#B and P#C

Page 7, last sentence in penultimate para: Says that the PCM will automatically erase codes after three consecutive tests

-- but it actually needs 40 warm

- up cycles.

Page 8, 2 nd para, 3 rd line: The word ‘both’ needs to be either moved or removed.

Page 8, 6 th line of section 2.5: Needs to include ‘psi’ as the unit for RVP

Page 12, last para: There are also some generic evaporative tr ouble codes in the P2 and P3 ranges.

Page 14, 4 th line: ‘does’ should read ‘do’

Page 21, Table 4.1: This table is not really necessary. It only repeats information already presented in Table 3.5