# NEW JERSEY MOTOR VEHICLE INSPECTION PROGRAM PIF EFFECTIVENESS STUDY

Final Report

Prepared for: 

## New Jersey Department of Environmental Protection Bureau of Motor Vehicle Inspection & Maintenance 401 East State Street Trenton, NJ 08625

Prepared by: MACTEC Engineering and Consulting, Inc. 21740 Beaumeade Circle, Suite 150 Ashburn, VA 20147 703.729.1416



### June 23, 2010NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report



## TABLE OF CONTENTS

1.0 INTRODUCTION ............................................................................................................ ................... 1 2.0 OTHER STATES’ PRACTICES ................................................................................................. ......... 3 3.0 ANALYTICAL APPROACH ...................................................................................................... ......... 4 4.0 ANALYTICAL RESULTS ....................................................................................................... ........... 7 4.1 Failure Rates for OBD Tests ............................................................................................ ........... 7 4.2 Proposed Discount Factors .............................................................................................. ........... 8 4.3 Clean Scanning / Clean Piping Analysis ................................................................................. . 10 5.0 RECOMMENDATIONS ........................................................................................................ .......... 13 5.1 Proposed Discount Factors .............................................................................................. ......... 13 5.2 Overall Network Effectiveness .......................................................................................... ....... 13

iNJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report


## 1.0 INTRODUCTION
In its original regulations on implementing an enhanced motor vehicle inspection program, the USEPA considered inspections performed at a d ecentralized test-and-repa ir garage to be only 50% as effective as those performed at a central ized test-only facilit y. Under pressure from states, the USEPA allowed inspection programs to claim higher degrees of decentralized testing effectiveness in their modeling as long as the claim was backed by sound reasoning. In New Jersey’s hybrid vehicle inspection network, Centraliz ed Inspection Facilities (CIFs) are test-only facilities run by the Parsons Group as a State contractor. Private Inspection Facilities (PIFs) are comprised of privately-owned test and repair facilities licensed by the State. The 

### New Jersey Department of Environmental Protection (the De partment or DEP) proposed an effectiveness of 80% for its decentralized PIFs and has been using th is factor in modeling for many years. When enhanced inspection was implemented in New Jersey, all vehicles received a tailpipe exhaust emissions test as their primary test. A tail pipe test may be subjec t to considerable fraud because it is easy to probe a known clean vehicl e (“clean piping”). While the Motor Vehicle Commission (MVC) aggressively pursues a nd prosecutes fraudulent inspection, a Sierra Research study in 2005, concluded th at a significant fraction of tailp ipe tests were either outright fraudulent or suspicious. With a large percentage of the motor vehicle fleet (model year 1996 and newer) now subject to On Board Diagnostics (OBD) testing as their pr imary emissions test, the landscape has changed. Newer OBD vehicles (2005+) have VINs embedded in the OBD data and even older OBD vehicles have a distinct “fi ngerprint” of OBD data that na rrows probable year/make/model/ engine family. As such, fraudulently substituting a known good vehicle for the vehicle under test (“clean scanning”) has beco me no less difficult, but far more easily detected. The MVC and DEP now have access to triggers and alerts when ever suspected fraudulent OBD testing occurs, virtually in real time. The Department considers 80% PIF effectiveness to likely be very conserva tive in light of the program and technology changes described abov e. In May 2010, the Department therefore authorized MACTEC to assess improvements in e ffectiveness of the decen tralized program. The purpose of this study is to determine a reasonable effectiveness fraction that may be supported by data and good technical reasoning. The data currently available for this study in cludes about ten years of enhanced inspection program data, plus limited roadside testing data . We evaluated the utility and relevance of the roadside data because it is not based on random sele ction of vehicles subject to inspection. It was determined that the use of this data could no t improve the statistical confidence of the larger dataset.

1NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report


### MACTEC analyzed the effectiven ess of the decentralized PIF network relative to the CIF (centralized) network. The relative effectiveness of PIFs is based on data collected from PIFs and CIFs in 2009. The final results of our evaluation are bei ng submitted by June 30, 2010, to coincide with program planning efforts that may be affected by the study outcome.

2NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report

3 

## 2.0 OTHER STATES’ PRACTICES
Since the decentralized portion of the New Jersey inspection program is not unlike decentralized testing in other States, MACTEC was directed to investigate relevant programs to determine if effectiveness rates in current us e are similar enough to indicate a standard for industry practice. MACTEC was able to determine that emi ssion inspection programs in Massachusetts, Connecticut, New York, Georgia an d Utah claim in their SIPs that their decentralized OBD inspections are as effective as cen tralized programs. N one of these States use the decentralized program effectiveness prompt of the MOBILE6.2 model when modeling the emission benefits for their OBD inspections. Although many states claim that their decentraliz ed tailpipe inspections are 100% effective, based on the supporting data included with this report, it seems more re asonable to keep New Jersey’s PIF effectiveness factor for tailpipe tests at 80%.NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report


## 3 .0 ANALYTICAL APPROACH
The following is a description of the analysis em ployed to re-calculate the PIF effectiveness for MOBILE6 modeling. It is broken down into the following categories:

• Previous program (CIF and PIF) data analysis

• Current program (CIF and PIF) data analysis

• M OBILE6.2 analysis This analysis defines the effectiveness of th e decentralized (PIF) ne twork relative to the centralized (CIF) network. CIFs are test-only facilities run by the Pa rsons Group as a State contractor. PIFs are comprised of privately-owned test and repair facilities licensed by the State.
Previous Program Data Analysis Protocol An indicator of PIF vs. CIF effectiveness is the incidence of clean scanning or clean piping. Clean piping and clean scanning refer to the practice of substituting a passing vehicle for the vehicle being tested. Clean piping occurs when an inspector probes th e tailpipe of a passing vehicle instead of the vehicle being tested. Clean scanning occurs when an inspector substitutes a fault-free vehicle for the vehicle that is being inspected. Previ ous program data from CIFs and PIFs was analyzed to determine rates of suspect ed fraud in PIFs and CIFs. From this, the effectiveness of PIF inspections is calculated relative to CIF inspections. This analysis focused on data collected during the la st two years of the old program. The following describes the possible indicators of fra ud that we calculated.
Suspected Clean Scanning Two parameters are calculated to determine if stations may have clean scanned a vehicle: 1. Mismatch between entered VIN and OBD VIN 2

. Questionable retests – mismatch between initial test monitors supported and retest monitors supported

Mismatch between entered VIN and OBD VIN -- If the vehicle has an electronic VIN available through the vehicle’s OBD II system , clean scanning cases can be identified by comparing the entered VIN with the VIN provide d by vehicle’s OBD II system. In order to perform this analysis, the data was manipulated to remove cases where mismatches between the OBDII VIN and the registered VI N may be in error. These mani pulations are presented below:

• Records where the OBDII VIN and the register ed VIN had less than 17 characters were removed.

• Records where the last digit of the OBDII VIN and the registered VIN was alpha (not numeric) were removed.

4NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report



### • M ismatches are based on the last 4 digits of the OBD VIN and the registered VIN. The percent of tests with VIN mismatches equals the following: (# of valid mismatches between OBD VIN/entered VIN)/(# of records with valid OBD VINs) Questionable Retests – Mismatch Between In itial Test Monitors Supported and Retest Monitors Supported -- The readiness monitors supported by the vehicle’s OBDII system provide a rough signature for the vehicle. Th e following analysis was conducted on OBDII inspection data to identify cases where readine ss monitors supported on an initial failing test do not match readiness monitors supported on passing retests: 1. Using data from the last two years, test reco rds for the initial failing test were identified (termed Fail Initial OBD ) 2. Records for vehicles that pass retests were identified (termed Pass OBD Retest ). 3. Data records for the Fail Initial OBD and Pass OBD Retest groups were linked via VIN. 4. Records where the monitor profile changes we re identified. Did not count cases where readiness status changes from r eady to not ready or vice versa. The percent of suspect rete sts equals the following: (# of monitor mismatches ) /

( # OBD retests ) Suspected Clean Piping Two parameters were calculated to determine if a station may have cl ean piped a vehicle: 1. Overall tailpipe failure rate 2. Questionable retests – less than 1⁄2 hour between initial fail ing test and pa ssing retest. Failure rate analysis – Initial test failure rates for the tailpip e test were calculated by model year and network (CIF vs. PIF). Then, using the stat ewide distribution of model years for the tailpipe test, normalized failure rates for tailpipe tests we re calculated for CIFs a nd PIFs. Rates for CIFs were compared to rates for PIFs. The relative failure rate is calculated as follows: (Normalized PIF failure rate)/(Normalized CIF failure rate)

Questionable Retests – Less than 1⁄2 Hour betw een Initial Failing Tailpipe Test and Passing Retest A short time (less than 1⁄2 hour) between an initial failing test and a passing retest could indicate that the vehicle was clean-pipe d on the retest. The following pr ocedure was used to identify vehicles that passed a retest less than 1⁄2 hour after they failed:

5NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report



### 1. Using data from the last two years, test reco rds for the initial failing test were identified (termed Fail Initial Tailpipe ) 2. Records for vehicles that pass retests were identified (termed Pass Tailpipe Retest ). 3. Data records for the Fail Initial Tailpipe and Pass Tailpipe Retest groups were linked via VIN. 4. Time between passing retests and failed initial test was calculated. 5. R ecords where passing retests occurred less than 1⁄2 hour after the failing test were identified. The percent of suspect rete sts equals the following: (# of passing retests performed less than 1⁄2 hour after failing test)/(# of tailpipe retests)

6NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report


## 4.0 ANALYTICAL RESULTS
This section provides comparativ e information on failure rates fo r OBD and tailpipe tests and then addresses the results from the cl ean piping and clean scanning analysis.

4.1 Failure Rates for OBD Tests Table 1 and Figure 1 compare OBD failure rates for CIFs with those for PIFs. The normalized fail rate for the PIFs is 98.5% of the fail rate for CIFs. As shown on Figure 1, PIFs have a highernormalized fail rate for readiness, while CIFs have a higher normalized fail rate for MILCommanded On. This makes sense, considering th at PIFs are much more likely than CIFs to attempt to correct a MIL situation before inspec ting the vehicle. This practice would lower MILon rates while raising not ready rates.

Table 1 – OBD Fail Rate: CIFs vs. PIFs

Model Year Weighting Fail Rate PIF CIF 1996 3% 14% 21% 1997 8% 14% 16% 1998 5% 14% 16% 1999 9% 11% 12% 2000 8% 11% 11% 2001 11% 14% 13% 2002 9% 10% 9% 2003 14% 7% 6% 2004 8% 6% 4% 2005 12% 4% 3% 2006 8% 3% 2% 2007 2% 2% 1% 2008 1% 0% 1% 2009 1% 0% 1% Normalized Fail Rate 8.8% 8.9%

7NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report
Figure 1- Readiness and MIL-On Fail Rate: CIFs vs. PIFs

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%

MIL-On Readiness Overall 

### Normalized OBDII Fail Rates: CIFs vs. PIFs

CIF PIF



### 4.2 Failure Rates for Tailpipe Tests Table 2 compares tailpipe failure rates for CIFs with those for PIFs. The normalized fail rate for the PIFs is 55% of the fail rate for CIFs. The lower normalized fail rate in PIFs can be partially explained by the common practice of repairing vehicles before their initial inspections.

8NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report
Table 2 – Tailpipe Fail Rate: CIFs vs. PIFs

Model Year Weighting Tailpipe Failure Rate PIFs CIFs Pre-81 1.85% 12.0% 1.9%

1981 0.19% 13.2% 0.2% 1982 0.25% 14.2% 0.3% 1983 0.37% 12.4% 0.4% 1984 0.75% 12.5% 0.8% 1985 1.10% 10.9% 1.1% 1986 1.91% 12.0% 1.9% 1987 2.52% 10.4% 2.5% 1988 3.44% 9.1% 3.4% 1989 3.99% 9.0% 4.0% 1990 5.38% 9.3% 5.4% 1991 5.72% 9.6% 5.7% 1992 8.41% 10.4% 8.4% 1993 10.74% 8.6% 10.7% 1994 15.24% 8.0% 15.2% 1995 18.62% 6.2% 18.6% 1996 0.73% 5.1% 0.7% 1997 1.10% 4.1% 1.1% 1998 1.15% 3.1% 1.2% 1999 1.27% 3.2% 1.3% 2000 1.57% 1.9% 1.6% 2001 1.66% 0.9% 1.7% 2002 1.63% 1.0% 1.6% 2003 1.93% 0.6% 1.9% 2004 2.68% 0.2% 2.7% 2005 3.05% 0.2% 3.0% 2006 1.32% 0.3% 1.3% 2007 0.69% 0.4% 0.7% 2008 0.53% 0.3% 0.5% 2009 0.15% 0.5% 0.2% 2010 0.02% 0.0% 0.0% Normalized Fail Rate 7.2% 13.1%

9NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report



### 4. 3 Clean Scanning / Clean Piping Analysis An indicator of PIF vs. CIF effectiveness is the incidence of clean scanning or clean piping. Clean piping and clean scanning refer to the practice of substituting a passing vehicle for the vehicle being tested. Clean piping occurs when an inspector probes th e tailpipe of a passing vehicle instead of the vehicle being tested. Clean scanning occurs when an inspector substitutes a fault-free vehicle for the vehicl e that is being inspected. Previous program data from CIFs and PIFs were analyzed to determine rates of suspected fraud in PIFs and CIFs. From this, the effectiveness of PIF inspections was calculated relative to CIF inspections. This analysis focused on data collected during the last year of the old program. Suspected Clean Scanning in OBD Inspections Two parameters were calculated to determine if stations may have clean scanned a vehicle: 1. Mismatch between entered VIN and OBD VIN 2

. Questionable retests – mismatch between initial test monitors supported and retest monitors supported
Mismatch between entered VIN and OBD VIN -- If the vehicle has an elec tronic VIN available through the vehicle’s OBD system, clean scanning cases can be identified by comparing entered VIN with the VIN provided by vehicle’s OBD system. In order to perform this analysis, the data were manipulated to remove cases where mismatches between the OBD VI N and registered VIN may be in error. These manipulations are presented below:

• Records where the OBD VIN and registered VI N had less than 17 characters were removed.

• Records where the last digits of the OBD VIN a nd registered VIN were alpha (not numeric) were removed.

• M ismatches were then manually reviewed to identify additional cases where the OBD VIN was invalid. Table 3 presents the number and percent of OBD VI N mismatches by network type. PIFs had a higher mismatch rate than CIFs, but the ov erall rate was low for both networks.

Table 3 – OBD VIN Mismatches: CIFs vs. PIFs Network # of Valid OBD VINs # of Valid Mismatches % with Valid Mismatch % Matching
CIF 318,488 559 0.18% 99.82%
PIF 27,278 207 0.76% 99.24%

The correct inspection rate at PIFs is 99.4% of th e correct inspection rate at CIFs, based on OBD VIN. Questionable Retests – Mismatch Between Initia l Test Monitors Supported and Retest Monitors Supported -- The readiness monitors supported by the vehi cle’s OBD system provide a rough signature
10NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report
for the vehicle. The following analysis was conducted on OBD inspection data to identify cases where readiness monitors supported on an initial failing test do not match readiness monitors supported on passing retests: 1. Using data from the last year, test records fo r the initial failing test were identified (termed Fail Initial OBD ) 2. Records for vehicles that pass retests were identified (termed Pass OBD Retest ). 3. Data records for the Fail Initial OBD and Pass OBD Retest groups were linked via VIN. 4

. Records where the monitor prof ile changes were identified. Table 4 presents the number and percent of monitor mismatches during retests by network type. Like OBD VIN mismatches, PIFs had a higher retest monitor mismatch rate than CIFs, but the overall rate was low for both networks.

Table 4 – Retest Monitor Mismatches: CIFs vs. PIFs

Network # Re-Inspections # with Monitor Mismatches % with Monitor Mismatches % without Monitor Mismatches
CIF 24,995 68 0.27% 99.73%
PIF 19,460 103 0.53% 99.47%

The correct re-inspection rate at PIFs is 99.7% of the correct re-inspection rate at CIFs, based on the percentage of retests with readiness monitor mismatches. Triggers Implemented with New Program The trigger incidence statistics reported above are based on data from 2009 where the State investigated mismatches in the following:

• OBD VIN

• PID Count/PCM Module ID Using the expanded OBD data collected by new equi pment in the PIFs, the State has implemented the following triggers, in addition to OBD VIN and PID Count/PCM Module ID mismatches:

• Communication Protocol mismatches

• Monitors supported mismatches

• Review of expanded PCM Module ID

• Review of CAL ID and CVN

• Extended data review: o Time since diagnostic trou ble codes were cleared. o D istance traveled with MIL on. The new triggers will further reduce the already low rates of fraudulent tests. 11NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report
Suspected Clean Piping in Tailpipe Inspections Due to time constraints for this task we were only able to analyze one parameter to determine if a station may have clean piped a vehicle: Questionable Retests – Less than 1⁄2 Hour between Initial Failing Tailpipe Test and Passing Retest The following procedure was used to identify vehicles that passed a retest less than 1⁄2 hour after they failed the initial test: 1. Using data from the last two years, test records for the initial failing test were identified (termed Fail Initial Tailpipe ) 2. Records for vehicles that passed retests were identified (termed Pass Tailpipe Retest ). 3. Data records for the Fail Initial Tailpipe and Pass Tailpipe Retest groups were linked via VIN. 4. The time between passing retests and failed initial test was calculated. 5. R ecords where passing retests occu rred less than 1⁄2 hour after the failing test were identified. A short time (less than 1⁄2 hour) between an initial failing test and a passing retest could indicate that the vehicle was clean-piped on the retest. However, in many cases tailpipe failures can be repaired in less than 1⁄2 hour. For example, a simple tune-up to co rrect high hydrocarbon emissions can conceivably be performed on some vehicles in less than 1⁄2 hour. Therefore, the percentage of tests identified by this trigger represents the upper limit of fraudulent tests. Table 5 presents the number and percent of tailpipe re tests that occurred less than 1⁄2 hour after the initial failing test. Even if there were no fraudulent retests, the percentage of passing retests within 1⁄2 hour of the initial failing test would be expected to be much lowe r in CIFs than in PIFs. As shown in Table 5, a higher percentage of passing retests occur within 1⁄2 hour of the initial failing test in PIFs than in CIFs. Overall, based on this trigger, the non-suspect re-insp ection rate at PIFs is 78% of the rate at CIFs.

Table 5 – Passing Reinspections That Occur Within 1⁄2 Hour of Initial Failing Tests: CIFs vs. PIFs

Network # Re-Inspections # Within 1/2 Hour Suspect ReInspection Rate Non-Suspect ReInspection Rate
CIF 19,791 1,212 6% 94%
PIF 53,578 14,360 27% 73%

12NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report


## 5.0 RECOMMENDATIONS
The recommendations presented in this section rela te first to the PIF effectiveness factors for OBD and tailpipe inspections and then for the overall network.
5 .1 Proposed Discount Factors
Recommendations on PIF effectiveness for OBD in spections and for tail pipe inspections are provided below. PIF Effectiveness for OBD Inspections MACTEC recommends that the State use a 100% effectiveness factor for OBD tests at PIFs, instead of the 80% factor that is currently use d. Our justification for increasing the effectiveness of OBD inspections at PIFs is:

• Fail rates for OBD inspections in PIFs ar e nearly identical to those in CIFs.

• Based on an analysis of triggers for OBD tests performed in 2009, over 99% of inspections in PIFs have no indications of fraud.

• Ne w Jersey has implemented several additiona l OBD triggers in the new program, which will further reduce the incidence of fraud. PIF Effectiveness for Tailpipe Inspections MACTEC recommends that the State continue to use an 80% effectiveness factor for tailpipe emissions tests at PIFs. The data analyzed in this task do not jus tify a higher or lower effectiveness factor. The fail rate in PIFs is 55% of the fail rate in CIFs, which can be partially explained by the common practice of repairing vehicles in PIFs be fore their initial inspections. The rate of non-suspect tests in PIFs is 78% of the rate in CI Fs, based on the percentage of passing retests that occur within 1⁄2 hour of the initial faili ng tests. Considering that the percentage of tests identified by this trigger represents the upper limit of fraudulent tests, the current PIF effectiveness factor of 80% is a conservative estimate. Further study would likely result in a higher PIF effectiveness factor.
5 .2 Overall Network Effectiveness Table 6 and Figure 2 show estimates of the overa ll network effectiveness based on the following data and assumptions:

• PIF effectiveness is as follows:

o OBD tests: 100%

o Tailpipe tests: 80%

13NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report



### • 30% of the vehicles are tested in PIFs. Based on historical data, the percent tested in PIFs is more like 20%, so this assumption results in a conservative, i.e., low, effectiveness factor for the overall network.

14NJ Motor Vehicle Inspection Program PI F Effectiveness Study June 23, 2010 MACTEC Engineering and Consu lting, Inc., Contract No: A61906 Final Report

15 Table 6 – Estimates of the I/M Program Effectiveness of the Network

Year PIF Testing OBD % PIF Testing TP % PIF I/M Effectiveness CIF I/M Effectiveness MOBILE6.2 I/M Effectiveness 2005 54.0% 46.0% 90.8% 100% 97.2% 2006 53.0% 47.0% 90.6% 100% 97.2% 2007 57.5% 42.5% 91.5% 100% 97.4% 2008 64.5% 35.5% 92.9% 100% 97.9% 2009 67.8% 32.2% 93.6% 100% 98.1% 2010* 71.7% 28.3% 94.3% 100% 98.3% 2011* 75.6% 24.4% 95.1% 100% 98.5% 2012* 79.5% 20.5% 95.9% 100% 98.8%

Figure 2 – Projected I/M Program Effectiveness

86% 88% 90% 92% 94% 96% 98% 100%

2005 2006 2007 2008 2009 2010* 2011* 2012* I/M Effectiveness by Year

PIF I/M Effectiveness Overall I/M Effectiveness