NewHampshire Motor Vehicle I/MSIP Revision

NewHampshire Departments of Safety and Environmental Services May 12, 2011

The State of New

Hampshire Motor Vehicle Inspection and Maintenance State Implementation Plan Revision

Attachment VIII : Quality Assurance Programming Documentation

Prepared by: The New

Hampshire Department of Environmental Services

Concord, NH November 1, 2011

~ DEPARTMENT OF 

# Environmental Services

Attachment VIII : Quality Assurance Programming Documentation 1New Hampshire Motor Vehicle I/MSIP Revision New Hampshire Departments of Safety and Environmental Services May 12, 2011 Quality Assurance Programming Documentation Table of Contents Quality Assurance ProgramElements .. . . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . . . .. . .. . . .. . . .. . . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . .. . . .. . . .. .3 OBDTriggers Analysis Protocol . . . .. . . .. . . . .. . .. . . .. . . .. . . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . .. . . . . .. . .. . . .. . . . .. . . .. . . .. . . .. . . .. . .. . . .. .3

Attachment VIII : Quality Assurance Programming Documentation 2New Hampshire Motor Vehicle I/MSIP Revision New Hampshire Departments of Safety and Environmental Services May 12, 2011 Quality Assurance ProgramElements The elements of I/M

quality assurance programs are outlined in 40 CFR 51 .363 and include : " Performance Audits, both overt and covert ; " Record Audits; " Equipment Audits; and " Auditor Training and Proficiency This Attachment to the I/M SIP submittal describes a vehicle test data analysis protocol (triggers analysis) used in NewHampshire in lieu of covert performance audits. DOS has worked with their OBD II vendor to develop and use sophisticated electronic analysis "triggers" to evaluate the performance of the decentralized inspection stations and inspectors by identifying anomalies and irregularities that might indicate fraud . Trigger analysis results available to NewHampshire DOS, fromthe automated I/M data system, enable DOS staff to efficiently and effectively pursue follow-up investigations and enforcement actions against stations and inspectors whose results appear to be outside expected parameters. To provide additional insight into this process, the triggers analysis is summarized below . Triggers-based data analyses/reports are not canned summary versions that are produced by the I/M data systemon a regular, periodic (e .g. monthly) basis. Instead, they are produced on an on-demand basis by DMV staff as needed to monitor inspections on an ongoing basis and to assist in investigations of specific stations or to search for particular patterns of potential violations or anomalies. New Hampshire will continue to evaluate incorporation of a covert auditing programin the state's I/M programin conjunction with the legislative OBDAdvisory Committee. Should such a programbe incorporated, DES will submit a SIP revision to reflect this change. At this time, however, it is the position of DMV and DES that the current triggers analysis is as, if not more, effective than a covert audit programutilizing vehicle set to fail . OBDTrigger Analysis Protocol The NHOST OBD Communication Protocol trigger compares the OBDtest results fromall the New Hampshire stations to those contained in a standard reference or "truth table" developed by the NHOST contractor, Gordon-Darby. Each OBD-equipped vehicle communicates with the NHOST test unit using one of five possible electronic OBDdata exchange protocols. These protocols are standards established by the Society of Automotive Engineers (SAE) for use by all the vehicle manufacturers. Each vehicle's OBDcommunications protocol is programmed into it at the time of manufacture and cannot be changed, making it possible to compare OBD protocol data collected during an NHOST test to the protocol identified as originally installed in the vehicle . Gordon-Darbyl developed and maintains a "truth table" that contains proper OBD protocols for families of vehicles subject to testing in the NHOST program . The table was built using test records collected in NewHampshire along with data fromGordon-Darby"s other vehicle inspection programs in Arizona, Kentucky, and Texas. These test data are analyzed to determine the reference communications protocol for each type of vehicle . 1 Gordon-Darby is recognized as an I/MProgramindustry expert in data collection and analysis . Gordon-Darby, with Sierra Research, developed a comprehensive triggers best practices guidance document for the United States Environmental Protection Agency (EPA) : "QA/QC Procedures Based on ProgramData and Statistical Process Control Methodsfor I/MPrograms," prepared for EPA, Certification and Compliance Division ; Sierra Research, Inc., Report No. SR01-10-02, October 2001 . Attachment VIII : Quality Assurance Programming Documentation 3New Hampshire Motor Vehicle I/MSIP Revision New Hampshire Departments of Safety and Environmental Services May 12, 2011 The NHOST OBD Communication Protocol trigger returns results showing the count and percent of tests performed by each station which have recorded protocols that (1) match those contained for each applicable VIN in the truth table, (2) do not match, and (3) are unknown. The last "unknown" group are the ones for which the truth table does not contain an identified protocol for that particular VIN. A test in which the recorded protocol does not match the type contained in the truth table is strong evidence that a vehicle other than the one that is supposed to be tested was fraudulently used in performing the OBDtest . This type of fraud is known as clean scanning, in which a vehicle that passes the OBDtest is tested in place of the vehicle actually subject to OBD inspection. Stations that have recorded a high count or high proportion of such protocol mismatches are likely involved in significant test fraud and are more closely scrutinized for enforcement action. In addition, there are two consumer hotlines available to report potential inspection station fraud . The DOS-DIVIV maintained consumer inquiry line is (603) 271-8800. Gordon-Darby maintains the consumer inquiry line (800) 295-5276 . Both of these phone lines are printed on each vehicle inspection report .

Attachment VIII : Quality Assurance Programming Documentation 4