Peer Review of Draft 

Report “Analysis of 

Evaporative On-Board

Diagnostic (OBD) 

Readiness and DTCs Using 

I/M Data”

February 24, 2014

Prepared for

Connie Hart

U.S. Environmental Protection Agency

Office of Transportation and Air Quality

2000 Traverwood Drive

Ann Arbor, Michigan 48105

Prepared by

Louis Browning

ICF International

620 Folsom Street, 2nd Floor

San Francisco, CA 94107
blank

page
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

ICF International i February 24, 2014

Contents

1. Introduction .............................................................................................................................1-1

2. Selection of Peer Reviewers ......................................................................................................2-1

3. Peer Review Process.................................................................................................................3-1

4. Summary of Review Comments.................................................................................................4-1

4.1. Responses to Charge Questions................................................................................................4-1

Appendix A. Resumes and Conflict of Interest Statements .............................................................. A-1

Appendix B. Charge Letter.............................................................................................................. B-1

Appendix C. Gene Tierney Review Comments................................................................................. C-1

Appendix D. Mike McCarthy Review Comments..............................................................................D-1

Appendix E. Michael St. Denis Review Comments............................................................................E-1

Tables

Table 2-1. EPA Suggested Reviewers.........................................................................................................2-1

Table 2-2. Final Reviewers .........................................................................................................................2-1
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

ICF International ii February 24, 2014

Acronyms and Abbreviations

Acronym / Abbreviation Stands For

CARB California Air Resources Board

DTC Diagnostic trouble code

EPA U.S. Environmental Protection Agency

ERG Eastern Research Group, Inc.

evap Evaporative emissions control system

HC Hydrocarbon

ICF ICF International

I/M or IM Inspection and Maintenance 

LDT Light-Duty Truck

LDV Light-Duty Vehicle

LEV Low Emission Vehicle

MY Model Year

OBD On-Board Diagnostics system.

SHED Sealed Housing for Evaporative Determination

WAM Work Assignment Manager
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Introduction

ICF International 1-1 February 24, 2014

1. Introduction

Gasoline vehicles are equipped with evaporative emissions control systems that control vapor from the 

fuel storage system while a vehicle is sitting or driving. When these systems or the vehicle's gasoline 

delivery system malfunction, excessive evaporative emissions can be emitted. Few estimates of the 

frequency of vehicles with evaporative emissions malfunctions, or leaks, in the fleet exist. These vehicles 

can have a significant impact on the hydrocarbon (HC) emissions inventory.

This report details the peer review of the subject draft report, Analysis of Evaporative On-Board 

Diagnostic (OBD) Readiness and DTCs Using I/M Data (December 11, 2013), which documents results 

from ERG's analysis of light-duty gasoline-powered vehicle On-board Diagnostic (OBD) evaporative 

emissions control system (evap) diagnostic trouble codes (DTCs) using inspection and maintenance (I/M) 

program data. A number of independent subject matter experts were identified and the process 

managed to provide reviews and comments on the new evaporative data analysis. This peer review 

process was carried out under EPA’s peer review guidelines

1

. 

This report is organized as follows:

 Chapter 2 details the selection of the peer reviewers

 Chapter 3 details the peer review process 

 Chapter 4 summarizes the reviews

 Appendix A provides resumes and conflict of interest statements for the three selected reviewers

 Appendix B provides the charge letter sent to the selected reviewers

 Appendix C, D and E provide the actual reviews submitted by the three selected reviewers

 

1 U.S. Environmental Protection Agency, Peer Review Handbook, 3rd Edition with appendices. Prepared for the U.S. EPA by 

Members of the Peer Review Advisory Group, for EPA’s Science Policy Council, EPA/100/B-06/002. Available at 

http://www.epa.gov/peerreview
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Introduction

ICF International 1-2 February 24, 2014 blank

page
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Selection of Peer Reviewers

ICF International 2-1 February 24, 2014

2. Selection of Peer Reviewers

The EPA WAM supplied a list of five reviewers that EPA determined would be capable of reviewing the 

subject report. They are listed in Table 2-1.

Table 2-1. EPA Suggested Reviewers

Reviewer Affiliation Results

Gene Tierney Opus Inspection

Systech - ESP

Had the necessary expertise and agreed to 

review the report

Mike McCarthy

California Air Resources Board

(acting as an independent 

contractor)

Had the necessary expertise and agreed to 

review the report

Michael St. Denis Revecorp. Had the necessary expertise and agreed to 

review the report

The three selected reviewers are listed in Table 2-2. Each had the necessary expertise, were available to 

review the report in a timely manner and had no conflict of interest. All were agreed upon by the EPA 

WAM.

Table 2-2. Final Reviewers

Reviewer Contact Information Necessary 

Expertise

Conflict of 

Interest

Gene Tierney

Opus Inspection (Systech – ESP)

765 Ahukini Street

Honolulu, Hawaii 96825

P: 202-340-7553

Gene.Tierney@OpusInspection.com

Yes No

Mike McCarthy

P.O. Box 8101

Mammoth Lakes, CA 93546

P: 626-771-3614

mmccarth@arb.ca.gov

Yes No

Michael St. Denis

Revecorp

5732 Lonetree Blvd

Rocklin, CA 95765

P: 916-786-1006

Michael@Revecorp.com

Yes No

Resumes and conflict of interest statements for the three reviewers can be found in Appendix A.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Selection of Peer Reviewers

ICF International 2-2 February 24, 2014 blank

page
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Peer Review Process

ICF International 3-1 February 24, 2014

3. Peer Review Process

Once the three reviewers had been decided upon and approved by the EPA WAM, a charge letter and 

the subject report were sent to each reviewer via secure email. Shortly after distributing the charge 

letter (see Appendix B) and supporting materials for the peer review, a teleconference was held 

between the selected peer reviewers, the EPA WAM, EPA-identified relevant project-related staff and 

ICF staff to clarify any questions the peer reviewers may have regarding the report/written materials. At 

the conference call, EPA provided technical and/or background information on the particular report 

under review.

During the review process, one reviewer had a question and the question and answer were distributed 

to all reviewers via email. Each reviewer provided a written peer review in a timely manner. These 

were sent to ICF who forwarded them directly to the EPA WAM.

ICF managed the peer review process to ensure that each peer reviewer had sufficient time to complete

their review of the data analysis by the deliverable data told to them (mid-January 2014). ICF adhered to 

the provisions of EPA’s Peer Review Handbook guidelines to ensure that all segments of the peer review

conformed to EPA peer review policy.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Peer Review Process

ICF International 3-2 February 24, 2014 blank

page
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Summary of Review Comments

ICF International 4-1 February 24, 2014

4. Summary of Review Comments

In this section, review comments from the three reviewers are summarized. Full comments can be 

found in Appendix C for Gene Tierney, Appendix D for Mike McCarthy and Appendix E for Michael St. 

Denis. Responses are summarized below relative to the charge questions. 

4.1. Responses to Charge Questions

The three reviewers provided comments on the subject report. They contained both general and 

specific comments. Many of the specific comments tended to be more editorial than content related 

and are not summarized here. General and specific comments that are content related are summarized 

here based upon subject. Editorial comments can be found in the individual reviews in Appendices C 

through E.

Project Goals

All three reviewers felt that the report did not adequately define the exact purpose of the project. The 

statement in the Introduction of the report, “The purpose of this Work Assignment (WA) is to perform 

analysis to better understand evap DTC rates for light-duty vehicles” is vague and does not get to the 

real purpose of the work. Mr. Tierney felt that there was not a hypothesis or research question raised. 

Mr. McCarthy felt that there was no suggestion of a current understanding of the work before the 

analyses were done. He also wanted specific questions or theories laid out. Mr. St. Denis stated that 

while the conclusions indicate data are used to represent evap DTC rates of the in-use fleet and the IM 

versus non-IM fleets, this was not made clear in the introduction.

Vehicles Included

Mr. McCarthy indicated that “enhanced evap” actually was phased in for the 95-98 MY vehicles in 

California and 96-99 MY vehicles federally. Mr. St. Denis and Mr. Tierney commented that the report 

only presents data on vehicles in the condition of being prepared for an I/M inspection and therefore 

does not represent the expected evap DTC rates of the operating fleet. Mr. St. Denis also noted that in 

California, vehicles that passed the ASM test were not failed even if they were “not ready”. Therefore 

the not ready rates for California are artificially high and the low evap DTC rates artificially low.

States Used

All three reviewers felt that the pretense of not identifying states was done poorly. All felt that the 

states were easily identifiable by the data presented in Table 1, particularly States A and D. It was not 

clear in any of the reviewers’ minds why this was done. Mr. St. Denis indicated that by hiding the states, 

it was not clear whether the methods used to process the data were correct. 
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Summary of Review Comments

ICF International 4-2 February 24, 2014

Pending DTCs

Mr. McCarthy points out that no state program collects pending code data because they are provided 

via Mode $07 of SAE J1979. States only collect Mode $03. He felt it unwise to include cases of DTCs 

present but no MIL command because they don’t necessarily indicate that the vehicle had a problem at 

the time of inspection. Mr. St. Denis indicated that pending DTCs could just be false positives.

Non-IM State

Mr. St. Denis indicated that State D, if Colorado, is not equivalent to a state without an I/M program. 

Colorado has a gas cap test and the testing provides advisory OBDII results. In addition, Colorado has 

and evap repair consumer assistance program which would lead to more repairs than a non-I/M state.

Not Ready Evap Monitors

Mr. Tierney indicates that the report does not explain why evap monitor unreadiness increase with age. 

He feels a hypothesis should be raised and tested. Mr. St. Denis indicated there is no justification to 

state that older vehicles could have a higher non-ready state. This could be a result of motorists 

disconnecting their batteries prior to an inspection to clear out potential DTCs or the result of a battery 

being disconnected during repair. Mr. McCarthy feels that the national percentage of evap not ready 

for initial inspection is not the same as what was found in IM programs.

Statistical Significance

Mr. Tierney felt that the report did not provide any statistical information on the number of vehicles in 

any of the cohorts. In addition, there was no discussion of statistical significance of the differences or 

similarities found. Mr. St. Denis thought there should be more analysis of different makes and models 

to see the difference in not ready and DTC rates. He noted that different environmental conditions can 

cause significant differences in evap emissions and cause the systems to function differently.

General

All three reviewers had many general comments correcting statements throughout the report. More 

details can be found in the three full reviews in Appendices C through E.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-1 February 24, 2014

Appendix A. Resumes and Conflict of Interest Statements
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-2 February 24, 2014

EUGENE J. TIERNEY

765 Ahukini Street

Honolulu, Hawaii 96825

202-340-7553

Gene.Tierney@OpusInspection.com

EXPERIENCE

Opus Inspection (Systech-ESP) I/M 

Consultant

Providing motor vehicle inspection services and support August 2011 to Present

U.S. Environmental Protection Agency

Office of Transportation and Air Quality

Washington, D.C. Senior Policy Advisor, 9/2006 – 7/2011

Acting Assistant Office Director, 2/2006 – 9/2006

Ann Arbor, Michigan Director, Center for Air Quality and Modeling, 1/2000 – 1/2006

Chief, Planning and Human Resources Staff, 2/1997 – 12/1999

Chief, I/M Section, 1/1987 – 1/1996

Environmental Scientist, 11/1979 – 12/1986

Key Responsibilities and Accomplishments

 Chaired Mobile Source Technical Review Subcommittee workgroups on modeling and 

Transitioning I/M. Prepared key reports on use of Remote OBD in I/M programs.

 Developed MOBILE6, NONROAD2004 and MOVES.

 Drove development of PEMS for use in research and compliance.

 Authored the IM240 test procedure and evaporative system pressure and purge tests.

 Responsible for development, implementation, and oversight of motor vehicle I/M programs. 

Wrote and promulgated rules and guidance pursuant to Clean Air Act requirements.

 Wrote numerous technical reports, briefings, policy papers, correspondence, quality control 

procedures, and equipment specs related to I/M and mobile source air pollution control.

 Prepared strategic inventories and developed new methods for creating and updating national 

emission inventories. Prepared an inventory for Cairo, Egypt. Assisted Shanghai, China in 

in-use data collection and analysis of mobile sources.

 Led ground-breaking, $4 million study of light-duty vehicle PM emissions in Kansas City. 

Received the EPA’s Science Achievement Award for this work.

 Prepared and delivered speeches at conferences and meetings. Testified before legislatures, 

city councils, and boards. Chaired committees and cross-organizational teams.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-3 February 24, 2014

Hawaii State Energy Office Ridesharing Coordinator

Department of Planning and Economic Development 1978 - 1979

Developed and implemented transportation energy conservation programs.

EDUCATION

UNIVERSITY OF MICHIGAN Master of Science

School of Natural Resources Resource Policy and Management

Ann Arbor, Michigan May 1982

UNIVERSITY OF HAWAII Bachelor of Arts, Environmental Studies

Honolulu, Hawaii December 1977

HONORS

Science Achievement Award

2009

OTAQ Diversity Award

2000

EPA Medals for Exceptional, Outstanding and Superior Service

Gold 1984

Silver 1993

Bronze 1992/2002
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-4 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-5 February 24, 2014

Mike McCarthy

Chief Technology Officer

Mobile Source Control Division

California Air Resources Board

mmccarth@arb.ca.gov

Currently, Mike has responsibility at ARB for assessment of vehicle and powertrain technology and 

future light-duty criteria pollutant and greenhouse gas emission standards including leading the review of 

the GHG standards already adopted out to the 2025 model year. Previously, Mike has worked on the onboard diagnostic (OBD) programs for ARB for over 18 years including managing all aspects of the lightduty and heavy-duty OBD programs. This included regulatory development, certification and 

implementation, in-use enforcement, and integration with inspection and maintenance programs 

nationwide as well as in California. 

Mr. McCarthy holds a B.S. degree in Mechanical Engineering with an emphasis in Digital Control 

Systems from the University of California, Los Angeles.

Chief Technology Officer, 11/2012- present

Emissions Compliance, Automotive Regulations, and Science Division

Air Resources Board

-Technical and program lead for existing and future light-duty vehicle greenhouse gas and criteria pollutant tailpipe 

standards including lead for review of national joint greenhouse gas and fuel economy regulations.

Manager, Advanced Engineering Section 2002-2012

Mobile Source Control Division

Air Resources Board

-Oversaw light-duty on-board diagnostics (OBD) program including regulatory updates, annual certification for all 

vehicles sold in the U.S. market, in-use enforcement including recall, and integration with OBD into inspection and 

maintenance programs nationwide and in California.

-Oversaw on-road heavy-duty OBD program from initial regulatory development through certification, and 

implementation in the 2010 through 2013 model years.

Engineer 1994-2002

Mobile Sources Control Division

Air Resources Board

-Worked as a lead staff in demonstrating technical feasibility for various OBD monitoring strategies, regulatory 

updates to the light-duty vehicle OBD regulation and associated guidance documents, and initiated an enforcement 

program to verify manufacturer’s OBD systems met the requirements that resulted in many recalls and settlements.

Test Engineer 1992-1994

Mobile Sources Control Division

Air Resources Board

-Worked as a test engineer to demonstrate technical feasibility for the first Low Emission Vehicle tailpipe standards 

with prototype emission controls installed on production vehicles.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-6 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-7 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-8 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-9 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-10 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-11 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-12 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-13 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-14 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-15 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-16 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-17 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-18 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-19 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-20 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Resumes and Conflict of Interest Statements

ICF International A-21 February 24, 2014 blank

page
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Charge Letter

ICF International B-1 February 24, 2014

Appendix B. Charge Letter

Peer Reviewer Charge

Charge to Peer Reviewers of “Analysis of Evaporative

On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Gasoline vehicles are equipped with evaporative emissions control systems that control vapor from the 

fuel storage system while a vehicle is sitting or driving. When these systems or the vehicle's gasoline 

delivery system malfunction, excessive evaporative emissions can be emitted. Few estimates of the 

frequency of vehicles with evaporative emissions malfunctions, or leaks, in the fleet exist. These vehicles 

can have a significant impact on the hydrocarbon (HC) emissions inventory.

This report pulls together five states of data for an analysis of the evaporative emissions related on-board 

diagnostics (OBD) codes.

You are asked to review and provide expert comments on the Analysis of Evaporative On-Board 

Diagnostic (OBD) Readiness and DTCs Using I/M Data.

In your comments you should distinguish between recommendations for clearly defined improvements 

that can be readily made based on data or literature reasonably available to EPA and improvements that 

are more exploratory or dependent on information not readily available to EPA. Your written comments 

should address all aspects of the report (methodologies, analysis, conclusions, and narrative) and should 

be sufficiently clear and detailed to allow readers to thoroughly understand their relevance to the subject 

report. In addition to addressing these issues, EPA encourages you to best apply your particular area(s) 

of expertise to review the overall study. Please deliver your final written comments to Lou Browning 

at ICF International (Louis.Browning@icfi.com) by January 15, 2014.

All materials provided to you as well as your comments should be treated as confidential, and should 

neither be released nor discussed with others outside of the review panel. Once EPA has made its 

reports and supporting documentation public, EPA will notify you that you may release or discuss the 

peer review materials and your review comments with others. 

If the reviewer has questions about what is required in order to complete this review or needs additional 

background material, please contact Lou Browning at ICF International (Louis.Browning@icfi.com or 831-

662-3683). If the reviewer has any questions about the EPA peer review process itself, please contact 

Ms. Ruth Schenk in EPA's Quality Office, National Vehicle and Fuel Emissions Laboratory 

(schenk.ruth@epa.gov or 734-214-4017).

Some specific areas of focus include the following:

1. Does the report meet its primary goal?

2. Is the description of analytic methods and procedures clear and detailed enough to allow the 

reader to develop an adequate understanding of the steps taken and assumptions made to 

develop the tables and figures in the report? Are examples selected for tables and figures wellchosen and designed to assist the reader in understanding the approach and methods?

3. Does the methodology, data, and analyses support the report's conclusion?
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Charge Letter

ICF International B-2 February 24, 2014 blank

page
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Gene Tierney Review Comments

ICF International C-1 February 24, 2014

Appendix C. Gene Tierney Review Comments

Gene Tierney’s Review of Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs 

Using I/M Data

I have completed my review of the draft report “Analysis of Evaporative On-Board Diagnostic (OBD) 

Readiness and DTCs Using I/M Data.” My comments and questions follow.

My first comment is that the purpose of this analysis is not at all clear. There does not seem to be a 

hypothesis or research question being tested or a potential application for the data analysis discussed. 

It would be helpful to be able to evaluate the results of the analysis against such a hypothesis, research 

question or proposed application. While the report does a good job of presenting the data and the 

analyses, it is difficult to determine whether the report meets its “primary goal” because that is not 

clearly stated.

My second comment is that two of the states in this analysis are clearly identifiable by the features of 

their programs (California and Colorado), so the pretense of not identifying them really doesn’t work! It 

is not clear what purpose is served by attempting to maintain anonymity of the programs.

Third, the report should address the uncertainties associated with the use of I/M data. Some motorists 

get repairs performed just prior to the initial inspection because the MIL is on and they know they will 

fail. Such repairs are generally followed by a clearing of codes and resetting of the monitors. So, the 

rate of monitor readiness detected at initial inspection is impacted by this phenomenon; this fact is 

bolstered by the decrease in unreadiness found when all inspections were considered. It is not known 

either how often pre-inspection repair occurs or how that might affect monitor readiness. As the report 

points out, this is a transitory issue. Again, depending on the ultimate use of this data analysis, this may 

or may not be a factor. The report mentions the idea of pursuing alternate non-I/M data sets to 

supplement this analysis; that would help address this issue.

Fourth, the report finds that evap monitor unreadiness appears to increase with age. It would be useful 

to put this into context: does readiness of other monitors change with age in a similar fashion or is just 

the evap monitor? If evap monitor unreadiness increases at a faster rate than other monitors this would 

be useful to understand. For comparison sake, a sample of the data for one other important monitor 

(e.g., catalyst) might sufficiently shed light on this question.

Fifth, on this same point, no discussion ensues about why evap monitor unreadiness increases with age. 

It might be useful to pose alternative hypotheses to explain this phenomenon that could lay the basis for 

future research programs. For example, one possibility is that the increase in evap monitor unreadiness 

has nothing to do with the evap system itself. We know that MIL on rates increase with age and, as a 

result of repair, the resetting of the OBD system. This kicks the monitors into a state of unreadiness. 
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Gene Tierney Review Comments

ICF International C-2 February 24, 2014

The evap monitor is harder to set so one would expect the frequency of evap monitor unreadiness to 

increase as a result. Alternately, is there something about evap monitoring systems that is impacted by 

age or deterioration such that it becomes increasingly difficult to get the evap monitor ready?

Sixth, the I/M programs in question all exempt vehicles from inspection for more than the first 2 years of 

the vehicles’ lives, yet the analyses show results for all states starting with vehicles at age 2 years old. 

This apparent contradiction should be explained in the text. I presume in the case of the California, the 

data is primarily for vehicles that change ownership prior to the 6 year old inspection requirement. This 

might also explain why in States B and C, the percent of inspections with monitors ready is apparently 

flat until after age 4 when the downward slope begins. Was consideration given to eliminating the preinspection requirement data? What impact does this have on the slopes of the lines in Figure 5 and 

others?

Seventh, the report does not provide any statistical information on the numbers of vehicles in any of the 

cohorts. No discussion is made of the statistical significance of the differences or similarities found. For 

example, in section 2.5, the report states that the slopes of the lines in Figure 5 for states A and D are 

different from states B and C. If they are different, the statistical basis for that statement should be 

elucidated. Likewise, after modifying the data, the slope for State B seems to fall in line with A and D 

but State C still appears to be different. In addition to discussing the statistical significance of these 

apparent differences it would be useful to address why State C does not seem to change. Is there 

something about the program design? The climate? There appears to be an anomaly but we don’t have 

any ideas as to why.

Eighth, Figure 11 compared to similar figures for the other states, appears to show that failure to require 

motorists to repair vehicles based on OBD results in significantly higher DTC rates. A key question for 

EPA as it continues to improve and upgrade the MOVES model is how well it characterizes non-I/M 

emission rates of the national fleet, especially for evap given its outsize contribution to the inventory. 

Further investigation of this finding may help shed light on the emission reduction benefits of OBD I/M 

with regard to evap. It appears that MOVES may underestimate evap deterioration outside I/M areas 

that enforce OBD. Such data may also be helpful for states in assessing the efficacy of the I/M program, 

as required by the Clean Air Act.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-1 February 24, 2014

Appendix D. Mike McCarthy Review Comments

Mike McCarthy’s Review of Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-2 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-3 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-4 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-5 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-6 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-7 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-8 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-9 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-10 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-11 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-12 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-13 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-14 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-15 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-16 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-17 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-18 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-19 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-20 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-21 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-22 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-23 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-24 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-25 February 24, 2014
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Mike McCarthy Review Comments

ICF International D-26 February 24, 2014 blank

page
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-1 February 24, 2014

Appendix E. Michael St. Denis Review Comments

Michael St. Denis review of EPA Draft Report 131211 – 1/25/2014

“Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

General Comments:

It is difficult to determine the exact purpose of this project based on the description “The purpose of 

this Work Assignment (WA) is to perform analysis to better understand evap DTC rates for light-duty 

vehicles.” The conclusions seem to indicate the data are to be used to represent the evap DTC rates of 

the in use fleet, and the IM versus non-IM fleets. However, because all of the data were collected from 

vehicles expecting an IM test, the data are not representative of the operating fleet during the entire 

operating time between IM inspections. 

Because the states are not identified (it is unclear why the states are not identified), it is difficult to 

determine if the methods used to process the data are correct. For instance, I believe State A is 

California. Based on conversations with CA BAR, they were not failing most vehicles for readiness if they 

passed the ASM test (1996 and newer vehicles received both tests). Therefore, the not ready rates for 

CA are artificially high because not ready vehicles were not required to get set to ready and retested. If 

the vehicles were not forced to be ready, this could cause artificially low evap DTC rates since evap is 

one of the later monitors to get set to ready. The data from CA would be more like State D data where 

OBDII was not enforced but advisory. (If state A is CA, the authors should contact Garrett Torgerson at 

CA BAR for clarification of how the readiness requirements were implements in the time period data 

was used). It would also be useful for the reader to know the states because there could be 

temperature, altitude or vehicle fleet longevity (average age) differences between the states which 

might impact the results. This is important since in some cases the data are combined or compared to 

each other.

The report indicates on page 6 (paragraph 1, line 5) that the evap DTC data were used regardless of if 

the MIL was commanded on. Therefore pending DTC data may be included. Pending DTC data are NOT 

an indication of a problem, but could be false positives. For this reason, only evap DTC where the MIL 

was commanded on should have been used in this analysis. The use of the pending DTC data is 

incorrect. 

The data for State D are referred to as “the non IM state”, are not equivalent in any way to data from an 

actual data for an operating fleet in a state without an IM program. The state does have a fuel cap test 

which is motorists fail they have to get repaired in order to pass their IM inspection. In addition, 

motorist do get advisory OBDII results, so if there is an evap problem the motorist is told about the 

problem and encouraged to repair it. If State D is Colorado, the state even has an evap repair consumer 

assistance program which would lead to more repairs that in a non IM case. I think the authors need to 
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-2 February 24, 2014

use great caution or not use the term “non-IM state” when referring to State D, and this makes some of 

the comparisons potentially invalid.

ERG should try to not refer to themselves in the third person as doing the work. Where it says “ERG 

tabulated” or ERG summed” etc., there is no reason to use “ERG”. Just say the data were tabulated or 

summed, etc.

Specific Comments:

Page: 5

Paragraph 1, Sentence 2 - This report ONLY presents data on vehicles in the condition of being prepared 

for an I/M inspection and that data does not represent the expected evap DTC rates of the operating 

fleet.

Paragraph 2, Sentence 4 "model year 1999"... - And CARB gave exemptions to some through 2000 I 

believe, you would need to check with Mike McCarthy at CARB.

Table 1, State A - If state A is CA, if a vehicle passed the ASM test and failed the OBDII test, was it 

allowed to pass the overall inspection? I believe at least back in 2004 this was the case and could have 

been through 2010.

Page: 6

Paragraph 1, line 5 - Pending DTCs may not be an indication of an actual problem, so including these in 

the analysis is inappropriate. Why was it not simply required that the MIL also be commanded on in this 

case?

Paragraph 4, line 4 - It is not clear what "for one inspection cycle" is referring to.

Paragraph 6, line 1 - Change from "inspection cycles" to "inspections" as this is too difficult to 

understand.

Page: 7

Paragraph 5, line 7 "higher incidence" - There is no justification to state that older vehicles could 

therefore have a "Higher" not ready rate. In use vehicles, unless the power is disconnected, should not 

have not ready evap monitors, and then only for a short period of time.

Page: 8

Paragraph 1, line 2 "The oldest IM Fleet..." - This should not be assumed. The average age of vehicles 

which were tested should be determined and presented in the table below if this is to be claimed (it 

could justify the comment above as well). State B had the shortest model year exemptions, but had the 

second highest not ready rate.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-3 February 24, 2014

Paragraph 2 - This logic is incorrect because it is known that motorists disconnect their battery to clear 

potential diagnostic trouble codes immediately prior to coming in for an inspection. That rate is not 

known, and must be accounted for. It is not clear when this states national percentage of "vehicles in IM 

programs" if this is referring to only vehicles going into an IM inspection.

Paragraph 3, lines 8 to the end - It is common during repairs to disconnect the battery of a vehicle when 

it is being repaired, so one would expect that the not ready rate of the evap monitor would be higher on 

subsequent tests. I believe overall not ready rates increase on retests in general, so this result seems 

counter intuitive.

Page: 9

Paragraph 1, lines 5 "although..." to the end of the paragraph - This does not read correctly. Once a 

monitor is set to ready it remains set to ready until there is a battery disconnect. Therefore the 

readiness monitor reporting not ready at a higher rate is quite possibly due to it taking longer to get 

ready but this is only after a battery disconnect which must have occurred to set the monitor to not 

ready.

Title 2.4 - I would use the term "Vehicle model year" or "vehicle age".

Figures 1 - 4 - The label "model_yr" should be cleaned up on the graphs on the bottom left and should 

say something like "Vehicle Model Year". The label "state = A" or B, C or D should be removed above the 

graph as it is in the title already.

Page: 14

Paragraph 2, line 2 - Why was the range 89% to 96% readiness chosen?

Paragraph 2, line 4 - Define "well outside". The 89% and 96% seem arbitrary, so saying a value is well 

outside this arbitrary line is meaningless.

Paragraph 3, line 7 and 8 - Why aren't the 2 year old State A vehicles are not representative of the fleet?

Paragraph 3, line 10 - Why would testing in an "off" I/M read cause the readiness to be lower? I would 

suspect that this is because these vehicles are being forced to come in for a test due to change of 

ownership and therefore are more likely to be tampered by disconnecting the battery.

Page: 15

Figure 5 - In figure 5, the rate of decrease in readiness with model year for states A and D are almost 

identical, but A is 5% lower.

Page: 17

Paragraph 1, lines 7 to 9 - Just because the number of inspections are lower, that should have no 

correlation on the rate of readiness.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-4 February 24, 2014

Paragraph 1, line 7 - These vehicles may not represent the "true fleet at that age" why and in what way?

Paragraph 2, line 4 and5 - This therefore biases the data to reflect the rate of having a not ready evap 

monitor ONLY during the short period in a two year cycle just prior to an IM test. The vehicles which 

were removed are still operating in the fleet, and if the results of this study are to identify the rate of 

evap monitors not ready in the operating fleet, then removing these vehicles is not acceptable.

Page: 20

Paragraph 1, lines 4 and 5 - As noted in the previous comment, if other vehicles are removed, this 

statement ONLY applies to vehicles in their IM inspection cycle.

Paragraph 3 at the end - The last part of this sentence "which is a very small percentage of the I/M 

fleet", should be removed as it does not add any quantitative information.

Last sentence on the page - This says "In general". What other manufacturer P codes were considered. 

Was it only P1093, P1094, etc. that exactly match the generic codes?

Page: 21

Sentence under table - This sentence should be set out, the way it is formatted it appears to be a 

continuation of the prior section.

3.1 first sentence - Replace with "For all vehicles with a “ready” evap monitor, the number of vehicle 

test records containing evap-related DTCs for each inspection cycle was determined by calendar year 

and model year.

Page: 22

Paragraph 1, line 1 and 2 - "Of the inspections performed" is confusing to the reader when it is changed 

from looking at the results "per cycle" and then this seems to say "per inspection".

Paragraph 1, line 4 - Should say "was not" calculated. Even if the monitor is not complete, there can still 

be some evap DTCs, they are just not necessarily all that could occur.

Paragraph 2, lines 2 and 3 - Since you have the data, determine this and present it as opposed to 

theorizing this could be the reason.

Table 5 footnote - How does this change if only model year 2000 and newer vehicles are considered? 

Maybe pre 1996, post 1996 and all should be shown in the table since through 2000 the vehicles may 

not have been compliant.

Page: 23

Figures 8 through 11 - Take off the "state = X" at the top of the graph since it is in the title. Fix the 

"model_yr" title at the bottom.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-5 February 24, 2014

Page: 27

Paragraph 2, first sentence - For State A there is an inflection point at 5 years (the rates increase 

significantly at year 6), but this could also somehow be attributed to the 6 model year exemption.

Paragraph 2, lines 3 and 4 - The rate of increase of the percentage of vehicle tests with evap DTCs 

increased the older the vehicle for State A.

Paragraph 3 - Setting arbitrary reference lines and then stating one set is generally below them and 

another is generally above them is not a good comparison - Figure 12 does a much better job.

Paragraph 4, second sentence - It should be noted here "therefore it is expected the failure rate would 

be higher because motorist are not required to repair vehicles with evap problems". And that the failure 

rate is twice as high as State A.

Page: 30

Paragraph 2, "might not be representative of the IM Fleet" line 1 - Once again, what is the purpose of 

this report? Is it to represent the evap DTC rate of vehicles "during IM" or in general? What is the 

hypothesis for "off cycle" inspections being non-representative and how could this be tested?

Figure 14 - Is there any theory as to why A and C are so different between the unmodified and modified, 

but B and D are not? There should be some discussion here for Figure 14.

Page: 33

Paragraph 1, line 3 "three states" - No, all states show that older vehicles are more likely to have stored 

evap DTCs.

Paragraph 2, line 10 - I am not sure "minimum" is the correct term here or at least it is confusing to me. I 

think you mean that it is the minimum evap DTC rate because although OBDII is not enforced, the area 

does have a fuel cap tests, therefore the evap DTC rate in an non IM area is expected to be higher.

Paragraph 3, line 6 - You MUST add "but there is a fuel cap inspection" so the reader is not mislead that 

this is the OBDII IM versus non-IM difference - which would be expected to be larger. It may be useful 

here to report the fuel cap failure rate in area D as a surrogate for the expected increase if the fuel cap 

was not inspected.

Page: 34

Paragraph 1, first sentence - Although it says it below, this needs to be caveated here with the fuel cap 

inspection info here is you make this strong of a statement or move this sentence to the next paragraph 

were this is discussed.

Page: 35

Figure 15 - Move Figure 15 up one page so it follows where it was mentioned.
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-6 February 24, 2014

Page: 36

Paragraph 1, line 2 - For some of the data and model years you could only see one cycle, but for others 

you could see several. This should be noted and it should be pointed out that this is weighted towards 

older vehicles which have been through more cycles.

Paragraph 2, lines 4 to 6 - The second sentence here does not really say what the results are showing, 

although the part in the parenthesis does say it. It should be clear that these results imply that the 

actual rate of vehicles with evap problems is most likely higher than the observed rate of vehicles with 

evap DTCs. Also, it is expected that some of the vehicles which were not ready could be in that state 

because the motorist intentionally disconnected the battery, resetting the readiness monitors and DTCs 

because the MIL was on and a DTC - possibly an evap DTC - was set and they were trying to get through 

inspection without it being detected.

Paragraph 3, line 1 - "presented" should be "presents"

Page: 37

Table 6 - The horizontal lines in Table 6 for State A need to be added.

Page: 38

Paragraph 1, line 1 - It should be noted that it has been shown in other analyses that different vehicle 

makes and models have different not ready and DTC rates. Therefore differences in fleet mix in the 

various states (and differences in vehicle age) could well be contributing to the differences by state. It 

should also be noted that environmental conditions (temperature, pressure, etc.) can cause significantly 

difference evap emissions and cause the systems to function differently. It would be useful to know the 

environmental conditions of each state, but that would give away at least two of the states.

Paragraph 2, line 2 - The word "usually" does not seem to belong here.

Table 7 -What is the difference between P_456 "small leak detected" and P_442 "small leak"? That just 

does not seem correct as they seems to be the same thing (you can't have a P_442 unless you detected 

it).

Page: 39

Paragraph 2, line 2 - I believe there is a significant problem with Ford vehicles which set this code and it 

causes the results to be biased. This should most likely be mentioned here.

Paragraph 4, line 1 - Insert the word "evap" prior to "monitor"

Page: 40

Paragraph 1, second sentence - I believe this statement is untrue, but it should be checked with Mike 

McCarthy. The evap monitor may not be totally complete so that it indicates "ready" but it could have 
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-7 February 24, 2014

run some parts of the monitor and found problems which are stored as DTCs prior to the monitor being 

totally complete.

Paragraph 1, last sentence - It should be noted, "even a state which does not perform pass/fail IM 

inspections, they are only advisory."

Paragraph 2, line 3 - "an evap monitor that is not ready" should be changed to "the evap monitor not 

ready", because there is only one evap monitor.

Paragraph 1, third sentence - This makes no sense. Disconnecting the battery sets ALL monitors to not 

ready. The second half of this sentence is the correct explanation for this observation.

Paragraph 4, first sentence - And vehicles in a non IM area do not have an increase in not ready evap 

monitors as they age? What did Sate D show?

Page: 42

Paragraph 1, last sentence - Should this also states that it can be used to estimate the minimum benefit 

of IM programs on reducing high evap emitting vehicles?

Page: 43

Paragraph 2, line 6 - This sentence starts with "also" but this sentence is not related to the sentences 

prior to it. It is important to note again here that there is a fuel cap test and that the OBDII test is 

performed, is advisory and therefore motorist could still be obtaining evap repairs based on the 

advisory.

Paragraph 4, last sentence - Why? How is this related or why is it suggested?

Page: 47

All B Tables - It is noted that readiness decreases with age, but for all States, the 1996 vehicles has high 

readiness rates. Is there any explanation for this?
Peer Review of Draft Report “Analysis of Evaporative On-Board Diagnostic (OBD) Readiness and DTCs Using I/M Data”

Michael St. Denis Review Comments

ICF International E-8 February 24, 2014 blank

page