# Peer Review of Draft Report “ Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data ”

February 2 4 , 2014

Prepared for Connie Hart U.S. Environmental Protection Agency Office of Transportation and Air Quality 2000 Traverwood Drive Ann Arbor, Michigan 48105
Prepared by Louis Browning ICF International 620 Folsom Street, 2 nd 

### Floor San Francisco , CA 94107Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

ICF International i February 24, 2014 

# Contents

1. Introduction

................................ ................................ ................................ ............................. 1

- 1
2. Selection of Peer Reviewers

................................ ................................ ................................ ...... 2

- 1
3. Peer Review Process

................................ ................................ ................................ ................. 3

- 1
4. Summary of Review Comments

................................ ................................ ................................ . 4

- 1 4.1. Responses to Charge Questions

................................ ................................ ................................ 4

- 1 Appendix A. Resumes and Conflict of Interest Statements

................................ .............................. A

- 1 Appendix B. Charge Letter

................................ ................................ ................................ .............. B

- 1 Appendix C. Gene Tierney Review Comments

................................ ................................ ................. C

- 1 Appendix D. Mike McCarthy Review Comments

................................ ................................ .............. D

- 1 Appendix E. Michael St. Denis Review Comments

................................ ................................ ............ E

- 1 

# Tables

Table 2

- 1. EPA Suggested Reviewers

................................ ................................ ................................ ......... 2

- 1 Table 2

- 2. Final Reviewers

................................ ................................ ................................ ......................... 2

- 1Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

ICF International ii February 24, 2014 

# Acronyms and Abbreviations

Acronym / Abbreviation Stands For

CARB California Air Resources Board

DTC Diagnostic trouble code

EPA U.S. Environmental Protection Agency

ERG Eastern Research Group , Inc.

evap Evaporative emissions control system

HC Hydrocarbon

ICF ICF International I/M or IM Inspection and Maintenance

LDT Light

- Duty Truck

LDV Light

- Duty Vehicle

LEV Low Emission Vehicle

MY Model Year

OBD On

- B oard Diagnostics system

. SHED Sealed Housing for Evaporative Determination

WAM Work Assignment ManagerPeer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Introduction

ICF International 1

- 1 February 24, 2014 

# 1. Introduction
Gasoline vehicles are equipped with evaporative emissions control systems that control vapor from the fuel storage system w hile a vehicle is sitting or driving. When these systems or the vehicle's gasoline delivery system malfunction, excessive evaporative emissions can be emitted. Few estimates of the frequency of vehicles with evaporative emissions malfunctions, or leaks, i n the fleet exist. These vehicles can have a significant impact on the hydrocarbon (HC) emissions inventory.

Th is report d etails the peer review of the subject draft report, Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data ( December 11, 2013) , which documents results from ERG's analysis of light

- duty gasoline

- powered vehicle On

- board Diagnos tic (OBD) evaporative emissions control system (evap) diagnostic trouble codes (DTCs) using inspection and maintenance (I/M) program data. A number of independent subject matter experts were identified and the process
manage d to provide review s and comment s on the new evaporative data analysis. This peer review process was carried out u n der EPA’s peer review guideline s 1



### . This report is organized as follows:

 

### Chapter 2 details the selection of the peer reviewers

 

### Chapter 3 details the peer review process

 

### Chapter 4 summarizes the reviews

 

### Appendix A provides resumes and conflict of interest statements for the three selected reviewers

 

### Appendix B provides the charge letter sent to the selected reviewers

 

### Appendix C , D and E pro vide the actual reviews submitted by the three selected reviewers

1 U.S. Environmental Protection Agency, Peer Review Handbook, 3rd Edition with appendices. Prepared for the U.S. EPA by Members of the Peer Review Advisory Group, for EPA’s Science Policy Council, EPA/100/B

- 06/002 . Available at http://www.epa.gov/peerreviewPeer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Introduction

ICF International 1

- 2 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Selection of Peer Reviewers

ICF International 2

- 1 February 24, 2014 

# 2. Selection of Peer Reviewers
The EPA WAM supplied a list of f ive reviewers that EPA determined would be capable of reviewing the subject report. They are listed in Table 2

- 1.

Table 2

- 1

. EPA Suggested Reviewers

Reviewer Affiliation Results

Gene Tierney Opus Inspection Systech

- ESP Had the necessary expertise and agreed to review the report

Mike McCarthy California Air Resources Board (acting as an independent contractor) Had the necessary expertise and agreed to review the report

Michael St. Denis Revecorp. Had the necessary expertise and agreed to review the report



### The three selected reviewers are listed in Table 2

- 2. Each had the necessary expertise, were available to review the report in a timely manner and had no conflict of interest. All were agreed upon by the EPA WAM.

Table 2

- 2

. Final Reviewers

Reviewer Contact Information Necessary Expertise Conflict of

Interest

Gene Tierney Opus Inspection (Systech

– ESP) 765 Ahukini Street Honolulu, Hawaii 96825 P: 202

- 340

- 7553 Gene.Tierney@OpusInspection.com Yes No

Mike McCarthy P.O. Box 8101 Mammoth Lakes, CA 93546 P: 626

- 771

- 3614 mmccarth@arb.ca.gov Yes No

Michael St. Denis Revecorp 5732 Lonetree Blvd Rocklin, CA 95765 P: 916

- 786

- 1006 Michael@Revecorp.com Yes No



### Resumes and conflict of interest statements for the three reviewers can be found in Appendix A.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diag nostic (OBD) Readiness and DTCs Using I/M Data” Selection of Peer Reviewers

ICF International 2

- 2 February 24, 2014Peer Revie w of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Peer Review Process

ICF International 3

- 1 February 24, 2014 

# 3. Peer Review Process
Once the three reviewers had been decided upon and approved by the EPA WAM, a charge letter and the subject report were sent to each reviewer via secure email. Shortly after distributing the charge letter (see Appendix B) and supporting materials for the peer review, a teleconference was held between th e selected peer reviewers , the EPA WAM, EPA

- identified relevant project

- related staff and ICF staff to clarify any questions the peer reviewers may have regarding the report/written materials. At the confer ence call, EPA provided technical and/or background information on the particular report under review

. During the review process, one reviewer had a question and the question and answer were distributed to all reviewers via email . Each reviewer provided a written peer review in a timely manner. These were sent to ICF who forwarded them directly to the EPA WAM.

ICF manage d the peer review process to ensure that each peer review er had sufficient time to complete their review of the data analysis by the deli verable data told to them ( mid

- January 201 4 ) . ICF adhered t o the provision s of EPA’s Peer Review Handbook guidelines to ensure that all segments of the peer review conformed to EPA peer review policy.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Peer Review Process

ICF International 3

- 2 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Summary of Review Comments

ICF International 4

- 1 February 24, 2014 

# 4. Summary of Review Comments

In this section, review comments from the three reviewers are summarized. Full comments can be found in Appendix C for Gene Tierney , Appendix D for Mike McCarthy and Appendix E for Michael St. Denis

. Responses are summarized below relative to the charge questions.



# 4.1. Responses to Charge Questions

The three reviewers provided comments on the subject report. They contained both general and specific comments. Many of the specific comments tended to be more editorial than content related and are not summarized here. General and specific comments that are content related are summarized here based upon subject. Editorial comments can be found in the individual reviews in Appendices C through E.


### Project Goals
All three reviewers felt that the report did not adequately define the exact purpose of the project. The statement in the Introduc tion of the report, “ The purpose of this Work Assignment (WA) is to perform analysis to better understand evap DTC rates for light

- duty vehicles ” is vague and does not get to the real purpose of the work. Mr. Tierney felt that there was not a hypothesis o r research question raised. Mr. McCarthy felt that there was no suggestion of a current understanding of the work before the analyses were done. He also wanted specific questions or theories laid out. Mr. St. Denis stated that while the conclusions indi cate data are used to represent evap DTC rates of the in

- use fleet and the IM

versus non

- IM fleets, this was not made clear in the introduction.



### Vehicles Included
Mr. McCarthy indicated that “enhanced evap” actually was phased in for the 95

- 98 MY vehicles in California and 96

- 99 MY vehicles federally. Mr. St. Denis and Mr. Tierney commented that the report only presents data on vehicles in the condition of being prepared for an I/M inspection and therefore does not represent the expected evap DTC rates of the operating fleet. Mr. St. Denis also noted that in California, vehicles that passed the ASM test were not failed even if they were “not ready”. Therefore the not ready rates for California are artificially high and the low evap DTC rates artificially low.



### States Used

All three reviewers felt that the pretense of not identifying states was done poorly. All felt that the states were easily identifiable by the data presented in Table 1, particularly States A and D. It was not cle ar in any of the reviewers’ minds why this was done. Mr. St. Denis indicated that by hiding the states, it was not clear whether the methods used to process the data were correct.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Usi ng I/M Data” Summary of Review Comments

ICF International 4

- 2 February 24, 2014 

### Pending DTCs
Mr. McCarthy points out that no state program collects pendi ng code data because they are provided via Mode $07 of SAE J1979. States only collect Mode $03. He felt it unwise to include cases of DTCs present but no MIL command because they don’t necessarily indicate that the vehicle had a problem at the time of in spection. Mr. St. Denis indicated that pending DTCs could just be false positives.



### Non

- IM State

Mr. St. Denis indicated that State D, if Colorado, is not equivalent to a state without an I/M program. Colorado has a gas cap test and the testing provides a dvisory OBDII results. In addition, Colorado has and evap repair consumer assistance program which would lead to more repairs than a non

- I/M state.


### Not Ready Evap Monitors
Mr. Tierney indicates that the report does not explain why evap monitor unreadiness increase with age. He feels a hypothesis should be raised and tested. Mr. St. Denis indicated there is no justification to state that older vehicles could have a higher non

- ready state. This could be a result of motorists disconnecting their batteries prior to an inspection to clear out potential DTCs or the result of a battery being disconnected during repair. Mr. McCarthy feels that the national percentage of evap not ready for initial inspection is not the same as what was found in IM programs.



### Sta tistical Significance

Mr. Tierney felt that the report did not provide any statistical information on the number of vehicles in any of the cohorts. In addition, there was no discussion of statistical significance of the differences or similarities found. Mr. St. Denis though t there shou ld be more analysis of different makes and models to see the difference in not ready and DTC rates. He noted that different environmental conditions can cause significant differences in evap emissions and cause the systems to function differently.



### General
All three reviewers had many general comments correcting statements throughout the report. More details can be found in the three full reviews in Appendices C through E.Peer Review of Draft R eport “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 1 February 24, 2014 

# Appendix A. Resumes and Conflict of Interest StatementsPeer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 2 February 24, 2014 

### EUGENE J. TIERNEY 765 Ahukini Street Honolulu, Hawaii 96825
202

- 340

- 7553 Gene.Tierney@OpusInspection.com

EXPERIENCE 

### Opus Inspection (Systech

- ESP) I/M Consultant Providing motor vehicle inspection services and support August 2011 to Present
U.S. Environmental Protection Agency Office of Transportation and Air Quality Washington, D.C. Senior Policy Advisor, 9/2006

– 7/2011 Acting Assistant Office Director, 2/2006

– 9/2006 Ann Arbor, Michigan Director, Center for Air Quality and Modeling, 1/2000

– 1/2006 Chief, Planning and Human Resources Staff, 2/1997

– 12/1999 Chief, I/M Section, 1/1987

– 1/1996 Environmental Scientist, 11/1979

– 12/1986 Key Responsibilities and Accomplishments

 Chaired Mobile Source Technical Review Subcommittee workgroups on modeling and Transitioning I/M. Prepared key reports on use of Remote OBD in I/M programs.

 Developed MOBILE6, NONROAD20 04 and MOVES.

 Drove development of PEMS for use in research and compliance.

 Authored the IM240 test procedure and evaporative system pressure and purge tests.

 Responsible for development, implementation, and oversight of motor vehicle I/M programs. Wrote and promulgated rules and guidance pursuant to Clean Air Act requirements.

 Wrote numerous technical reports, briefings, policy papers, correspondence, quality control procedures, and equipment specs related to I/M and mobile source air pollution control.

 P repared strategic inventories and developed new methods for creating and updating national emission inventories. Prepared an inventory for Cairo, Egypt. Assisted Shanghai, China in in

- use data collection and analysis of mobile sources.

 Led ground

- breakin g, $4 million study of light

- duty vehicle PM emissions in Kansas City. Received the EPA’s Science Achievement Award for this work.

 Prepared and delivered speeches at conferences and meetings. Testified before legislatures, city councils, and boards. Chair ed committees and cross

- organizational teams.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readine ss and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 3 February 24, 2014 

### Hawaii State Energy Office Ridesharing Coordinator Department of Planning and Economic Development 1978

- 1979 Developed and implemented transportation energy conservation programs.

EDUCATION



### UNIVERSITY OF MIC HIGAN Master of Science School of Natural Resources Resource Policy and Management Ann Arbor, Michigan May 1982

UNIVERSITY OF HAWAII Bachelor of Arts, Environmental Studies Honolulu, Hawaii December 1977

HONORS



### Science Achievement Award

2009
OTAQ Diversity Award 2000 EPA Medals for Exceptional, Outstanding and Superior Service

Gold 1984 Silver 1993 Bronze 1992/2002Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 4 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Stat ements

ICF International A

- 5 February 24, 2014 

### Mike McCarthy Chief Technology Officer Mobile Source Control Division California Air Resources Board mmccarth@arb.ca.gov
Currently, Mike has responsibility at ARB for assessment of vehicle and powertrain technology and future light

- duty criteria pollutant and greenhouse gas emission standards including leading the review of the GHG standards already adopted out to the 2025 m odel year. Previously, Mike has worked on the on

board diagnostic (OBD) programs for ARB for over 18 years including managing all aspects of the light

duty and heavy

- duty OBD programs. This included regulatory development, certification and implementatio n, in

- use enforcement, and integration with inspection and maintenance programs nationwide as well as in California.
Mr. McCarthy holds a B.S. degree in Mechanical Engineering with an emphasis in Digital Control Systems from the University of California , Los Angeles.
Chief Technology Officer, 11/2012

- present Emissions Compliance, Automotive Regulations, and Science Division Air Resources Board

- Technical and program lead for existing and future light

- duty vehicle greenhouse gas and criteria poll utant tailpipe standards including lead for review of national joint greenhouse gas and fuel economy regulations.
Manager, Advanced Engineering Section 2002

- 2012 Mobile Source Control Division Air Resources Board

- Oversaw light

- duty on

- board diagnosti cs (OBD) program including regulatory updates, annual certification for all vehicles sold in the U.S. market, in

- use enforcement including recall, and integration with OBD into inspection and maintenance programs nationwide and in California.

- Oversaw on

- r oad heavy

- duty OBD program from initial regulatory development through certification, and implementation in the 2010 through 2013 model years.
Engineer 1994

- 2002 Mobile Sources Control Division Air Resources Board

- Worked as a lead staff in demon strating technical feasibility for various OBD monitoring strategies, regulatory updates to the light

- duty vehicle OBD regulation and associated guidance documents, and initiated an enforcement program to verify manufacturer’s OBD systems met the requireme nts that resulted in many recalls and settlements.
Test Engineer 1992

- 1994 Mobile Sources Control Division Air Resources Board

- Worked as a test engineer to demonstrate technical feasibility for the first Low Emission Vehicle tailpipe standards wi th prototype emission controls installed on production vehicles.Peer Review of Draft Report “Analysis of Evaporative O n

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 6 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 7 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resum es and Conflict of Interest Statements

ICF International A

- 8 February 24, 2014Peer Review of Draft R eport “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 9 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 10 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Read iness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 11 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 12 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Intere st Statements

ICF International A

- 13 February 24, 2014Peer Review of Draft Report “Analysis of Evap orative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 14 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 15 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I /M Data” Resumes and Conflict of Interest Statements

ICF International A

- 16 February 24, 2014Peer Re view of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 17 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 18 Febru ary 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagn ostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 19 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Conflict of Interest Statements

ICF International A

- 20 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Resumes and Co nflict of Interest Statements

ICF International A

- 21 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Boar d Diagnostic (OBD) Readiness and DTCs Using I/M Data” Charge Letter

ICF International B

- 1 February 24, 2014



# Appendix B. Charge Letter

Peer Reviewer Charge Charge to Peer Reviewers of “ Analysis of Evaporative On

- Board Diagnostic (OBD) R eadiness and DTCs Using I/M Data ”

Gasoline vehicles are equipped with evaporative emissions control systems that control vapor from the fuel storage system while a vehicle is sitting or driving. When these systems or the vehicle's gasoline delivery system malfunction, excessive evaporative emissions can be emitted. Few estimates of the frequency of vehicles with evaporative emissions malfunctions, or leaks, in the fleet exist. These vehicles can have a significant impact on the hydrocarbon (HC) emissions inventory. This report pulls together five states of data for an analysis of the evaporative emissions related on

- board diagnostics (OBD) codes. You are asked to review and provide expert comments on the Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data. In your comments you should distinguish between recommendations for clearly defined improvements that can be readily made based on data or literature reasonably available to EPA and improvements that are more exploratory or dependent on information not readily available to EPA. Your written comments should address all aspects of the report (methodologies, analysis, conclusions, and narrative) and should be sufficiently clear and detai led to allow readers to thoroughly understand their relevance to the subject report. In addition to addressing these issues, EPA encourages you to best apply your particular area(s) of expertise to review the overall study

. Please deliver your final writte n comments to Lou Browning at ICF International ( Louis.Browning@icfi.com ) by January 15, 2014. All materials provided to you as well as your comments should be treated as confidential, and should neither be re leased nor discussed with others outside of the review panel. Once EPA has made its reports and supporting documentation public, EPA will notify you that you may release or discuss the peer review materials and your review comments with others.
If the rev iewer has questions about what is required in order to complete this review or needs additional background material, please contact Lou Browning at ICF International ( Louis.Browning@icfi.com or 831

- 662

- 3683). If the reviewer has any questions about the EPA peer review process itself, please contact Ms. Ruth Schenk in EPA's Quality Office, National Vehicle and Fuel Emissions Laboratory ( schenk.ruth@ep a.gov or 734

- 214

- 4017). Some specific areas of focus include the following:

1. Does the report meet its primary goal?
2. Is the description of analytic methods and procedures clear and detailed enough to allow the reader to develop an adequate understanding of t he steps taken and assumptions made to develop the tables and figures in the report? Are examples selected for tables and figures well

chosen and designed to assist the reader in understanding the approach and methods?
3. Does the methodology, data, and analy ses support the report's conclusion?Peer Review of Draft Rep ort “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Charge Letter

ICF International B

- 2 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Gene Tierney Review Comments

ICF International C

- 1 February 24, 2014 

# Appendix C. Gene Tierney Review Comments

Gene Tierney’s Review o f Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data
I have completed my review of the draft report “ Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data .” My comments and questions follow.

My first comment is that the purpose of this analysis is not at all clear. There does not seem to be a hypothesis or research question being tested or a potential application for the data analysis discussed. It would be helpful to be able to evaluate the results of the analysis against such a hypothesis, research question or proposed application. While the report does a good job of pre senting the data and the analyses, it is difficult to determine whether the report meets its “primary goal” because that is not clearly stated.

My second comment is that two of the states in this analysis are clearly identifiable by the features of their p rograms (California and Colorado), so the pretense of not identifying them really doesn’t work! It is not clear what purpose is served by attempting to maintain anonymity of the programs.

Third, the report should address the uncertainties associated with the use of I/M data. Some motorists get repairs performed just prior to the initial inspection because the MIL is on and they know they will fail. Such repairs are generally followed by a clearing of codes and resetting of the monitors. So, the rate of monitor readiness detected at initial inspection is impacted by this phenomenon; this fact is bolstered by the decrease in unreadiness found when all inspections were considered. It is not known either how often pre

- inspection repair occurs or how that mi ght affect monitor readiness. As the report points out, this is a transitory issue. Again, depending on the ultimate use of this data analysis, this may or may not be a factor. The report mentions the idea of pursuing alternate non

- I/M data sets to
supp lement this analysis; that would help address this issue.

Fourth, the report finds that evap monitor unreadiness appears to increase with age. It would be useful to put this into context: does readiness of other monitors change with age in a similar fash ion or is just the evap monitor? If evap monitor unreadiness increases at a faster rate than other monitors this would be useful to understand. For comparison sake, a sample of the data for one other important monitor (e.g., catalyst) might sufficiently shed light on this question.

Fifth, on this same point, no discussion ensues about why evap monitor unreadiness increases with age. It might be useful to pose alternative hypotheses to explain this phenomenon that could lay the basis for future research pr ograms. For example, one possibility is that the increase in evap monitor unreadiness has nothing to do with the evap system itself. We know that MIL on rates increase with age and, as a result of repair, the resetting of the OBD system. This kicks the monitors into a state of unreadiness.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Gene Tierney Review Comments

ICF International C

- 2 February 24, 2014 

### The evap monitor is harder to set so one would expect the frequency of evap monitor unreadiness to increase as a result. Alternately, is there something about evap monitoring systems that is impacted by age or deteri oration such that it becomes increasingly difficult to get the evap monitor ready?
Sixth, the I/M programs in question all exempt vehicles from inspection for more than the first 2 years of the vehicles’ lives, yet the analyses show results for all states starting with vehicles at age 2 years old. This apparent contradiction should be explained in the text. I presume in the case of the California, the data is primarily for vehicles that change ownership prior to the 6 year old inspection requirement. Thi s might also explain why in States B and C, the percent of inspections with monitors ready is apparently flat until after age 4 when the downward slope begins. Was consideration given to eliminating the pre

inspection requirement data? What impact does t his have on the slopes of the lines in Figure 5 and others?

Seventh, the report does not provide any statistical information on the numbers of vehicles in any of the cohorts. No discussion is made of the statistical significance of the differences or simi larities found. For example, in section 2.5, the report states that the slopes of the lines in Figure 5 for states A and D are different from states B and C. If they are different, the statistical basis for that statement should be elucidated. Likewise, after modifying the data, the slope for State B seems to fall in line with A and D but State C still appears to be different. In addition to discussing the statistical significance of these apparent differences it would be useful to address why State C d oes not seem to change. Is there something about the program design? The climate? There appears to be an anomaly but we don’t have any ideas as to why.

Eighth, Figure 11 compared to similar figures for the other states, appears to show that failure to r equire motorists to repair vehicles based on OBD results in significantly higher DTC rates. A key question for EPA as it continues to improve and upgrade the MOVES model is how well it characterizes non

- I/M emission rates of the national fleet, especially for evap given its outsize contribution to the inventory. Further investigation of this finding may help shed light on the emission reduction benefits of OBD I/M with regard to evap. It appears that MOVES may underestimate evap deterioration outside I/M areas that enforce OBD. Such data may also be helpful for states in assessing the efficacy of the I/M program, as required by the Clean Air Act.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTC s Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 1 February 24, 2014



# Appendix D. Mike McCarthy Review Comments

Mike McCarthy’s Review o f Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M DataPeer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 2 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 3 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 4 Feb ruary 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) R eadiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 5 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 6 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 7 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Revie w Comments

ICF International D

- 8 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Di agnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 9 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 10 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 11 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Dat a” Mike McCarthy Review Comments

ICF International D

- 12 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 13 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 14 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 15 February 24 , 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readines s and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 16 February 24, 2014Peer Review of Dr aft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 17 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 18 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 19 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board D iagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 20 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 21 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 22 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M D ata” Mike McCarthy Review Comments

ICF International D

- 23 February 24, 2014Peer Review of Draft Report “Analysis o f Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 24 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 25 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Mike McCarthy Review Comments

ICF International D

- 26 February 24, 2014Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Rea diness and DTCs Using I/M Data” Michael St. Denis Review Comments

ICF International E

- 1 February 24, 2014 

# Appendix E. Michae l St. Denis Review Comments

Michael St. Denis review of EPA Draft Report 131211

– 1/25/2014 “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

General Comments: It is difficult to determine the exact purpose of this project based on the description “The purpose of this Work Assignment (WA) is to perform analysis to better understand evap DTC rates for light

- duty vehicles.” The conclusions seem to indicate the dat a are to be used to represent the evap DTC rates of the in use fleet, and the IM versus non

- IM fleets. However, because all of the data were collected from vehicles expecting an IM test, the data are not representative of the operating fleet during the en tire operating time between IM inspections.

Because the states are not identified (it is unclear why the states are not identified), it is difficult to determine if the methods used to process the data are correct. For instance, I believe State A is Cal ifornia. Based on conversations with CA BAR, they were not failing most vehicles for readiness if they passed the ASM test (1996 and newer vehicles received both tests). Therefore, the not ready rates for CA are artificially high because not ready vehicl es were not required to get set to ready and retested. If the vehicles were not forced to be ready, this could cause artificially low evap DTC rates since evap is one of the later monitors to get set to ready. The data from CA would be more like State D data where OBDII was not enforced but advisory. (If state A is CA, the authors should contact Garrett Torgerson at CA BAR for clarification of how the readiness requirements were implements in the time period data was used). It would also be useful for t he reader to know the states because there could be temperature, altitude or vehicle fleet longevity (average age) differences between the states which might impact the results. This is important since in some cases the data are combined or compared to

ea ch other.

The report indicates on page 6 (paragraph 1, line 5) that the evap DTC data were used regardless of if the MIL was commanded on. Therefore pending DTC data may be included. Pending DTC data are NOT an indication of a problem, but could be false positives. For this reason, only evap DTC where the MIL was commanded on should have been used in this analysis. The use of the pending DTC data is incorrect.

T he data for State D are referred to as “ the non IM state ” , are not equivalent in any way to data from an actual data for an operating fleet in a state without an IM program. The state does have a fuel cap test which is motorists fail they have to get repaired in order to pass their IM inspection. In addition, motorist do get advisory OBDII res ults , so if there is an evap problem the motorist is told about the problem and encouraged to repair it. If State D is Colorado, the state even has an evap repair consumer assistance program which would lead to more repairs that in a non IM case. I think the authors need toPeer R eview of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Michael St. Denis Review Comments

ICF International E

- 2 February 24, 2014 

### use great caution or not use the term “non

- IM state” when referring to State D, and this makes some of the comparisons potentially invalid.

ERG should try to not refer to themselves in the third person as doing the work. Where it says “ERG tabulated” or ERG summed” etc., there is no reason to use “ERG”. Just say the data were tabulated or summed, etc.

Specific Comments: Page: 5 Paragraph 1, Sentence 2

- This report ONLY presents data on vehicles in the condition of being prepared for an I/M inspection and that data does not represent the expected evap DTC rates of the operating fleet.
Paragraph 2, Sentence 4 "model year 1999"...

- And CARB gave exemptions to some through 2000 I believe, you would need to check with Mike McCarthy at CARB.
Table 1, State A

- If state A is CA, if a vehicle passed the ASM test and failed the OBDII test, was it allowed to pass the overall inspection? I believe at least back in 2004 this was the case and could have been through 2010.

Page: 6 Paragraph 1, line 5

- Pending DTCs may not be an indication of an actual problem, so including these in the analysis is inappropriate. Why was it not simply required that the MIL also be commanded on in this case?
Paragraph 4, line 4

- It is not clear what "for one inspection cycle" is referring to.
Paragraph 6, line 1

- Change from "inspection cycles" to "inspections" as this is too difficult to understand.

Page: 7

P aragraph 5, line 7 "higher incidence"

- There is no justification to state that older vehicles could therefore have a "Higher" not ready rate. In use vehicles, unless the power is disconnected, should not have not ready evap monitors, and then only for a s hort period of time.

Page: 8 Paragraph 1, line 2 "The oldest IM Fleet..."

- This should not be assumed. The average age of vehicles which were tested should be determined and presented in the table below if this is to be claimed (it could justify the comme nt above as well). State B had the shortest model year exemptions, but had the second highest not ready rate.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Michael St. Denis Review Comments

ICF International E

- 3 February 24, 2014 

### Paragraph 2

- This logic is incorrect because it is known that motorists disconnect their battery to clear potential diagnostic trouble codes imme diately prior to coming in for an inspection. That rate is not known, and must be accounted for. It is not clear when this states national percentage of "vehicles in IM programs" if this is referring to only vehicles going into an IM inspection.
Paragraph 3, lines 8 to the end

- It is common during repairs to disconnect the battery of a vehicle when it is being repaired, so one would expect that the not ready rate of the evap monitor would be higher on subsequent tests. I believe overall not ready rates inc rease on retests in general, so this result seems counter intuitive.
Page: 9 Paragraph 1, lines 5 "although..." to the end of the paragraph

- This does not read correctly. Once a monitor is set to ready it remains set to ready until there is a battery disc onnect. Therefore the readiness monitor reporting not ready at a higher rate is quite possibly due to it taking longer to get ready but this is only after a battery disconnect which must have occurred to set the monitor to not ready.

Title 2.4

- I would us e the term "Vehicle model year" or "vehicle age".
Figures 1

- 4

- The label "model_yr" should be cleaned up on the graphs on the bottom left and should say something like "Vehicle Model Year". The label "state = A" or B, C or D should be removed above the graph as it is in the title already.

Page: 14 Paragraph 2, line 2

- Why was the range 89% to 96% readiness chosen?
Paragraph 2, line 4

- Define "well outside". The 89% and 96% seem arbitrary, so saying a value is well outside this arbitrary line is meaningless.
Paragraph 3, line 7 and 8

- Why aren't the 2 year old State A vehicles are not representative of the fleet?
Paragraph 3, line 10

- Why would testing in an "off" I/M read cause the readiness to be lower? I would suspect that this is because the se vehicles are being forced to come in for a test due to change of ownership and therefore are more likely to be tampered by disconnecting the battery.

Page: 15 Figure 5

- In figure 5, the rate of decrease in readiness with model year for states A and D a re almost identical, but A is 5% lower.

Page: 17 Paragraph 1, lines 7 to 9

- Just because the number of inspections are lower, that should have no correlation on the rate of readiness.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Da ta” Michael St. Denis Review Comments

ICF International E

- 4 February 24, 2014 

### Paragraph 1, line 7

- These vehicles may not represent the "true fleet at that age" why and in what way?
Paragraph 2, line 4 and5

- This therefore biases the data to reflect the rate of having a not ready evap monitor ONLY during the short period in a two year cycle just prior to an IM test. The vehicles which were removed ar e still operating in the fleet, and if the results of this study are to identify the rate of evap monitors not ready in the operating fleet, then removing these vehicles is not acceptable.

Page: 20 Paragraph 1, lines 4 and 5

- As noted in the previous comm ent, if other vehicles are removed, this statement ONLY applies to vehicles in their IM inspection cycle.
Paragraph 3 at the end

- The last part of this sentence "which is a very small percentage of the I/M fleet", should be removed as it does not add any quantitative information.
Last sentence on the page

- This says "In general". What other manufacturer P codes were considered. Was it only P1093, P1094, etc. that exactly match the generic codes?

Page: 21
Sentence under table

- This sentence should be set out, the way it is formatted it appears to be a continuation of the prior section.

3.1 first sentence

- Replace with "For all vehicles with a “ready” evap monitor, the number of vehicle test records containing evap

- related DTCs for each inspection cycle wa s determined by calendar year and model year.

Page: 22 Paragraph 1, line 1 and 2

- "Of the inspections performed" is confusing to the reader when it is changed from looking at the results "per cycle" and then this seems to say "per inspection".
Paragraph 1 , line 4

- Should say "was not" calculated. Even if the monitor is not complete, there can still be some evap DTCs, they are just not necessarily all that could occur.
Paragraph 2, lines 2 and 3

- Since you have the data, determine this and present it as o pposed to theorizing this could be the reason.

Table 5 footnote

- How does this change if only model year 2000 and newer vehicles are considered? Maybe pre 1996, post 1996 and all should be shown in the table since through 2000 the vehicles may not have be en compliant.

Page: 23 Figures 8 through 11

- Take off the "state = X" at the top of the graph since it is in the title. Fix the "model_yr" title at the bottom.Peer Review of Draft Report “Analy sis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Michael St. Denis Review Comments

ICF International E

- 5 February 24, 2014 

### Page: 27 Paragraph 2, first sentence

- For State A there is an inflection point at 5 years (the rates increase significantly at year 6), but this could also somehow be attributed to the 6 model year exemption.
Paragraph 2, lines 3 and 4

- The rate of increase of the percentage of vehicle tests with evap DTCs increased the older the vehicle for State A.
Paragraph 3

- Setting arbitrary reference lines and then stating one set is generally below them and another is generally above them is not a good comparison

- Figure 12 does a much better job.
Paragraph 4, second sentence

- It should be noted here "therefore it is expected the failure rate would be higher because motorist are not required to repair vehicles with evap problems". And that the failure rate is twice as high as State A.

Page: 30

Para graph 2, "might not be representative of the IM Fleet" line 1

- Once again, what is the purpose of this report? Is it to represent the evap DTC rate of vehicles "during IM" or in general? What is the hypothesis for "off cycle" inspections being non

- represe ntative and how could this be tested?
Figure 14

- Is there any theory as to why A and C are so different between the unmodified and modified, but B and D are not? There should be some discussion here for Figure 14.

Page: 33 Paragraph 1, line 3 "three state s"

- No, all states show that older vehicles are more likely to have stored evap DTCs.
Paragraph 2, line 10

- I am not sure "minimum" is the correct term here or at least it is confusing to me. I think you mean that it is the minimum evap DTC rate because although OBDII is not enforced, the area does have a fuel cap tests, therefore the evap DTC rate in an non IM area is expected to be higher.
Paragraph 3, line 6

- You MUST add "but there is a fuel cap inspection" so the reader is not mislead that this is t he OBDII IM versus non

- IM difference

- which would be expected to be larger. It may be useful here to report the fuel cap failure rate in area D as a surrogate for the expected increase if the fuel cap was not inspected.

Page: 34 Paragraph 1, first sentenc e

- Although it says it below, this needs to be caveated here with the fuel cap inspection info here is you make this strong of a statement or move this sentence to the next paragraph were this is discussed.

Page: 35 Figure 15

- Move Figure 15 up one page so it follows where it was mentioned.Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Michael St. Denis Review Comments

ICF International E

- 6 February 24, 2014 

### Page: 36 Paragraph 1, line 2

- For some of the data and model years you could only see one cycle, but for others you could see several. This should be noted and it should be pointed out that this is weighted towards ol der vehicles which have been through more cycles.
Paragraph 2, lines 4 to 6

- The second sentence here does not really say what the results are showing, although the part in the parenthesis does say it. It should be clear that these results imply that the actual rate of vehicles with evap problems is most likely higher than the observed rate of vehicles with evap DTCs. Also, it is expected that some of the vehicles which were not ready could be in that state because the motorist intentionally disconnected t he battery, resetting the readiness monitors and DTCs because the MIL was on and a DTC

- possibly an evap DTC

- was set and they were trying to get through inspection without it being detected.
Paragraph 3, line 1

- "presented" should be "presents"

Page: 3 7
Table 6

- The horizontal lines in Table 6 for State A need to be added.

Page: 38 Paragraph 1, line 1

- It should be noted that it has been shown in other analyses that different vehicle makes and models have different not ready and DTC rates. Therefore d ifferences in fleet mix in the various states (and differences in vehicle age) could well be contributing to the differences by state. It should also be noted that environmental conditions (temperature, pressure, etc.) can cause significantly difference ev ap emissions and cause the systems to function differently. It would be useful to know the environmental conditions of each state, but that would give away at least two of the states.
Paragraph 2, line 2

- The word "usually" does not seem to belong here.

T able 7

- What is the difference between P_456 "small leak detected" and P_442 "small leak"? That just does not seem correct as they seems to be the same thing (you can't have a P_442 unless you detected it).

Page: 39 Paragraph 2, line 2

- I believe there is a significant problem with Ford vehicles which set this code and it causes the results to be biased. This should most likely be mentioned here.
Paragraph 4, line 1

- Insert the word "evap" prior to "monitor"

Page: 40 Paragraph 1, second sentence

- I belie ve this statement is untrue, but it should be checked with Mike McCarthy. The evap monitor may not be totally complete so that it indicates "ready" but it could havePeer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Michael St. Denis Review Comments

ICF International E

- 7 February 24, 2014 

### run some parts of the monitor and found problems which are stored as DTCs prior to the mon itor being totally complete.
Paragraph 1, last sentence

- It should be noted, "even a state which does not perform pass/fail IM inspections, they are only advisory."
Paragraph 2, line 3

- "an evap monitor that is not ready" should be changed to "the evap monitor not ready", because there is only one evap monitor.
Paragraph 1, third sentence

- This makes no sense. Disconnecting the battery sets ALL monitors to not ready. The second half of this sentence is the correct explanation for this observation.

Para graph 4, first sentence

- And vehicles in a non IM area do not have an increase in not ready evap monitors as they age? What did Sate D show?

Page: 42 Paragraph 1, last sentence

- Should this also states that it can be used to estimate the minimum benefit of IM programs on reducing high evap emitting vehicles?

Page: 43 Paragraph 2, line 6

- This sentence starts with "also" but this sentence is not related to the sentences prior to it. It is important to note again here that there is a fuel cap test and that the OBDII test is performed, is advisory and therefore motorist could still be obtaining evap repairs based on the advisory.
Paragraph 4, last sentence

- Why? How is this related or why is it suggested?

Page: 47
All B Tables

- It is noted that readiness d ecreases with age, but for all States, the 1996 vehicles has high readiness rates. Is there any explanation for this?Peer Review of Draft Report “Analysis of Evaporative On

- Board Diagnostic (OBD) Readiness and DTCs Using I/M Data” Michael St. Denis Review Comments

ICF International E

- 8 February 24, 2014