vehicle which does not meet emission-related requirements constitutes a
violation of program regulations.

Incomplete Test

Incomplete tests may also lead to false passes. When there are un-
derhood tampering checks, 1f station personnel leave the hood of a vehicle
closed, this part of the inspection could not have been done. Similarly,
1f the gas cap 1s not removed, or if the inspector does not look under the
car, fuel filler inlet and catalyst checks could not have been performed.
Rpm requirements may be circumvented by omitting use of a tachometer con-
nection. Such an omission 1s, of course, easier with a manual analyzer
or an automated analyzer which does not lock out improper rpm levels out
of an official I/M test. Incomplete tests can be determined with either
passing or failing vehicles. However, as with investigating test omis-
sions, there are advantages to using failing vehicles, i.e., direct obser-
vation of the inspection 1s probably not necessary.

Other False Passes

False passes may also be caused by inadequate understanding of cost
waiver procedures. (Obviously if no cost waivers are possible under a
particular program's regulations, this situation will not be a problem.)
Cost waivers usually require a minimum amount of work to be done on the
vehicle, and the program management would therefore have to devise ways
to demonstrate that required work was not performed. Failing vehicles
would have to be introduced in order to simulate a cost waiver situation.

“pre-repair" of failing vehicles may also generate a false pass.
Vehicles which fail may be adjusted by station personnel and then be re-
corded as passing. This situation may even occur 1n programs with auto-
mated analyzers. In these programs, the analyzer normally has a computer-
ized "master menu," which allows a station inspector to choose either a
formal I/M test test mode or a manual emission test mode. If station
inspectors wish to check a vehicle's emissions prior to an official test,
they may choose the manual mode to see if the vehicle passes or fails.
If they find that the vehicle fails, they may simply perform the adjust-
ments the vehicle needs to pass, and then initiate the official test.

Badly maintained probes may leak, thereby diluting the sample and
causing a false pass. Automatic leak checks and flow checks, especially
on automated instruments, may minimize this phenomenon. However, to the
extent that leaky probes contribute to a dilution of a sample, additional
false passes will decrease the real failure rate.*

Manual analyzers exacerbate QA problems in a decentralized program,
because the pass/fail decision is left up to the individual inspector/
mechanic. This situation could lead to either a false pass or a false
fail decision (see below). There are no comprehensive studies or reports

* Badly maintained probes may of course also be identified through
formal periodic QA audits as well.


(but not close to the nearly 100% “hit" percentage reported by some pro-
grams). However, California (like New York) does attempt to minimize the
time taken per case, and reserves administrative and disciplinary actions.
Out of over 1400 audits, actions of this sort accounted for just over 13%
of the total. Office conferences, a much less time consuming action, were

held for more than twice as many cases.


CHAPTER 4

EXAMPLES OF RESULTS FROM CURRENT AUDITING PROGRAMS

As discussed above, the goal of an enforcement-oriented QA covert
auditing program is to deter and document program practices detrimental
to the I/M program. Education 1s an essential part of that deterrence.
It is difficult though to define what constitutes effectiveness of a
covert auditing program quantitatively.

Ultimately a covert auditing program is effective if it reduces the
incidence of detrimental program practices. Data on the prevalence of
different types of program problems in decentralized programs is general-
ly not available. It 1s, therefore, impossible to present an analysis of
the effect of different types of covert auditing programs. One measure
of effectiveness that some programs use in percentage of hits:

Where:
H 1s the percentage of hits;
h 1s the number of investigations where investigators
found what they were expecting to find; and
I is the total number of investigations.

In some programs, H approaches 100%. Percentage of hits as a measure of
effectiveness though can be rather misleading. First, it may be based on
both safety and I/M requirements in combined programs. If a safety
program requires very detailed test procedures and the covert auditors are
very strict, then it 1s relatively easy to find that the station is not
carrying out a full test procedure. Second, to the degree that auditors
perform audits on a "problem station" basis, they maximize the chance that
H will be 100%. Third, as programs start to use measures which are only
indicators of possible problems, H may be expected to decrease. There
are, therefore, many potential factors which affect the percentage of
hits, and it 1s useful to investigate other ways to determine the effec-
tiveness of I/M programs.

PREVALENCE OF DETRIMENTAL PRACTICES

The first step to defining how effective a covert auditing program
1s, 1s to define the prevalence of the practices the auditing is designed
to control. Table 4-1 provides summary data from the Colorado program.


Similarly, if a program plans to investigate stations' work on 1981
and later vehicles, then late model vehicles must obviously be utilized.
(This approach is important for testing late model maintenance practices
and tampering/fuel switching checks.) Setting up 1981 and later vehicles
to fail 1s difficult, and may involve modifications which would prevent
borrowing such vehicles.

Setting up Vehicles for Audits

How a vehicle should be set up again depends on what the audit will
be testing. There are only two requirements: that the vehicle clearly
be what it 1s supposed to be (1.e., a failing test car does in fact
fail), and that the program can prove it. The amount of proof necessary
will vary from state to state and area to area.

Setting up vehicles to pass still requires access to an emission ana-
lyzer which 1s regularly calibrated. However, scrutiny of the analyzer
and of the emission levels of audit vehicles themselves can be expected to
increase as the use of failing vehicles increases and stations are cited
for passing failing cars. More than anything else, emission levels and
particularly the pass/fail condition of vehicles must be repeatable. Work
undertaken by the Colorado Department of Health has shown that this is
particularly difficult for 1981 and later vehicles.* (It is worth noting
that 1f a hollowed out catalyst 1s used on an audit car, the engine sound
from the car may seem slightly abnormal.)

Setting up pre-1981 vehicles either to pass or fail is easier because
they are more adjustable. However, if a program wishes to check what was
done to a vehicle, then there must be very carefully documented verifica-
tions of the vehicle's condition. California performs such verifications,
not only for I/M, but also as a part of its covert audits to enforce con-
sumer protection laws on vehicle repairs. In California, a vehicle's con-
dition is thoroughly documented before 1t is sent to a station. For exam-
ple, to check whether spark plugs are actually charged, new plugs will be
gapped, photographed, aged, and installed prior to a covert investigation.
After the covert visit, the documentation process 1s repeated. It should
be noted this approach is not necessarily the model for all covert pro-
grams; the degree of documentation necessary 1s dependent on what 1s
being investigated and what 1s needed for successful disposition of cases
and actions resulting from audits.

Shop Facilities and Tools

Shops for vehicle preparation can be quite simple. The basic requir-
ement is an enclosed space where work may be performed on cars, preferably
with room for a few tools and an emission analyzer. Some of the more spe-
cialized tools for working on 1981 and later vehicles are shown in Figure
A-1 of Appendix A.

* See Schroeer, John, “Covert Vehicle Modification," Colorado Department
of Health, Denver, Colorado, 1985.

5-5


TABLE 1.2

CONCEPTS OF COVERT AUDITING

Alternative
1

Alternative
2

Observations of
Inspections by
Auditors

Necessary Training
of Auditors

vehicles Set-Up to
Fa1l/Pass

Shop Facilities

Method of Documen-
tation of Results

Direct; must actually
observe conduct of
inspection

Thorough knowledge of
regulations; automo-
tive background help-
ful

Pass

Generally not necessary
except for service fac-
ility with analyzer
which is periodically
audited and calibrated

Auditor makes a report
and will testify in
enforcement hearings

Only direct observations
necessary may be for
specific tampering
checks, e.g., fuel
filler inlet restrictors

Automotive training op-
tional; opens up auditor
jobs to general public

which may help to main-
tain cover of operations

Fail

May range from minimal
to extensive depending
on the range of investi-
gation areas and the
sophistication of the
set-up

Documentation, though
supplemented by auditor,
performed primarily by
shop personnel



Group 2:

Colum:

w

# Certs by Innacurate Prev Emis Tst Reslts

Indicates the number of certificates issued where the A test emission
results do not indicate a reduction of emissions when canpared with the

T or B test. This 13 calculated by comparing the actual emission readings
of the first test for a vehicle, tu the last test emssion readings Cor
the same vehicle. Tf they do not shew a reduction and a cectificate was
issued the count is incremental by one.

- & Dev from Statewide Dilution Average

Tnis field 1s used to determine 1f a imchanic is diluting the exhaust sample
by partially removing the probe from the exhaust pipe or inducing Leaks
elsewhere. A baseline dilution average for the district is determine! by
averaging the sum of the CO and C02 readings. The station and mechanic
dilution average 1s also calculated in the same manner.

The sta / mec dilution average is divided by the district average and a
percent deviation is calculated.

Idle Speed Deviation I to A Test (% pos) (8% dev)

The purpose is to see if mechanics are increasing idle speeds close to
thresholds instead of factory specifications to gain emissions reductions.
The idle speed of the first test for a vehicle is compared to the idle speed
of the next sequential test for the same vehicle. The percent they deviate
fram each other is calculated. The first figure in this colum lists the
percent of tests that had an increase in idle speed. The second figure shows
the percent those tests deviated fran the first test.

Number of Retests Adj OO to Threshold

The purpose is to indicate if mechanics are adjusting carburetor mixture
close to the OO threshold instead of factory specifications.

The first and next sequential test for a vehicle are compared to each other.
If the vehicle failed for CO emissions, the second test is checked to see
how close the idle CO adjustment brought the CO emission result to the OO
threshold. If the 00 emission was brought to within .5 of the ® threshold
a counter is increased by one. The number on the report is the total of
that counter.

Percent Tests W/In 113 RPM of Threshold

The idle speed 1s checked for each test to see if it comes within 113 REM
of the threshold for the specific emission standards category that the
vehicle falls into. The total mumber of tests that fit this pattern are

divided by the total number of tests and a percent is calculated and listed
in this colum.

Number A Tests With No Matching I Tests

This 1s to see 1f mechanics are perfonning A tests without an I or B test
This could indicate the previous emission readings may be ficticous.

Each A test is checked to see if it has a corresponding T or B test. TF m
matching test can be found a count er is incremented le Figure in thi.
colurn represents the nuiber of tines this wis the case.



APPENDIX B-1

COVERT VEHICLE INVESTIGATION REPORT FORMS
. (COLORADO AND NEW YORK)

Note - New York and Colorado also require a narrative
along with these forms where appropriate.


EES ENGINEERING -SCIENCE

Figure A.1 Sample Tools for an Audit Set-Up Shop

GM computer scanner and volt/ohm tester

Hand-operated vacuum pump, vacuum-pressure tester, various GM computerized

carburetor adjusting tools, and a distributor hold-down wrench



APPENDIX B-2

COVERT VEHICLE INVESTIGATION PROCEDURES
(CALIFORNIA AND NEW YORK)


OPERATION CODE
97
| t sreens [Ao PON oe | REMARKS
FRONT =|? SMES, [50 MMEELE OB | 88 SEL RSE
END 3 KING Pinson [Sf RINGS AND
Bate Jowmrs |! swocks Shacnces
| Tae Sz ome um Too
eresan ane
S service 53 EMERGENCY 101 Hoses vALvE:
Geane Seane svevew ino ces
2
| BRAKES | © Muttteterg. |°* Sturnesson [0% ates
7 orums/oiscs [SS Cacivens 103 States
B waren S56 am 104
Pump CONDITIONER Hosts
] COOLING [7 "3Mzefoue [27 weave 10° rweawostar
10 paciaton cam [58 neater nose | 106 cone
11 vatves, rock. /S9. 107
ERS OR HEADS SHORT BLOCK CONTROLS:
a ca Lc
ENGINE 13 CRANKCAsE 61 CAMSHAFT 109 CRANKSMAFT
CONTROL OR LIFTERS OR BEARING
1 14 wearer nose [62 wey gee 110 of pune :
15 carsureror 63 TANK OR wt _
OR MANIFOLD Lines BLOWER
FUEL 16 iis 84 ficrens 112 Tuecrors
| 17 evar, sys. [65 ain TS ies
16 pPLucs on 66 oistrieuroR 114 Insrauments:
wires OR MAGNETO OR SWITCHES
- 1 con 67 AEEUNATOR OF [TES CCETS oan =
] ELECTRICAL 68 116 wiRinc on
- sarreny stanten Connecrons
21 oisrr.cap 69 noron T17 conte &
Gay
2 cuuren 70 reansttse [tte Teheetroe
| cWwICH | See a i
TRANS. TRANSMISSION. DRIVE LINE U-JOUNTS.
24 Faunsutssion [72 cevennon 120 coos
| muurnee|’* caseere”—|*?" wantrouo
EXHAUST " ra weaven/tan | $22, txnausr svar,
CONTROL PIPES LACKETS:
ar 75 searing 123 wreee
“AXLE SHAFTS: OR SEALS CYLINDER.
J] REAR snoces | rennwnins [2 SER
AXLE 29 SPRINGS 7 THE-RODS. 125 Pins oR
Se saenues ‘Ano ENDS StnTs
30 tines 78 rorsion oars 1126
A
qT 3st ‘79 GLASS oR 127
oon aecuLaToRs werat
2 21
CAB MIRRORS 80 WS WIPERS 126 UPHOLSTERY.
| BODY 33 wre sce |! 129 saane
FRAME = | 54 (unr Sienberan oO sree | 5° paser :
35 63 131
| 36 cry 132
Puses worors cyLinpeas
3
HYDRAULIC [37 vauves 85 Mose umes [12> feet
38 186 134 =
| Cylinder ft) 2) 3) 4
Wall Taper 3) 6)-— 7) 8)
Crankshaft = [1) BE) 2) 4)
Rod Throws [5) 6) 7) 8)
Crankshaft ) 2) 3) 4)
a 5 > z DETAIL TASK CODE
" Drum/Dise. FRONT REAR 1 MARK & | 2 MARK & |3 MARK &
Measurement L- | ADJUST REPAIR REPLACE
\ 4 5 6
MEASURE TORQUE
INSTALL

Form 771-24 (11-74)


APPENDIX A

SHOP GUIDANCE

Facilities

A beginning shop should be large enough to house at least one vehicle
so that all work may be done inside. In addition, the shop should be e-
quipped with mechanics' tools (see below), an emission analyzer, supplies,
a roll-around tool box, a workbench and reference material. Space for an
office and a storage area 1s also desirable.

The shop should ideally have two bays, but a one-bay shop can still
be sufficient. A car hoist is useful, but if there is no hoist, the shop
should be equipped with a floor jack and car stands. An air compressor
can be located either outside or inside the shop area.

Tools

See Table A-1 and Figure A-1 for suggested tools for a basic shop.

A-1


FIGURE 2.2

CONCEPTUAL QA FRAMEWORK FOR ENFORCEMENT-ORIENTED COVERT AUDITS

IDENTIFIED
PROGRAM
PRACTICES

EVALUATION OF
SERIOUSNESS OF
POTENTIAL VIOLATIONS, ETC.
y

COMPLAINTS, TIPS, AND
SPECIFIC STATIONS SCREENED
FROM COMPUTERIZED DATA BASES

y
AVAILABLE ‘ PRIORITIZED STATIONS FOR
PERSONNEL COVERT INVESTIGATIONS

ENFORCEMENT ACTIONS

DETERRENCE
AND EDUCATION



CHAPTER 5

PROGRAM ISSUES

Chapters 1, 2, and 3 have discussed why covert auditing is important
and how 1t can be focused towards both enforcement and data collection.
Chapter 4 discussed results to date of several current auditing programs.
The purpose of this chapter is to describe specific program elements, and
to discuss how different model programs can be assembled.

PROGRAM ELEMENTS
Among the decisions programs will have to make will be:

o how to maintain the cover of auditors

eo whether to orient the covert program to passing
or failing cars

o what kind of people to hire as auditors and how
to organize a team

o what kinds of vehicles to procure and where to
procure them

o how to set up vehicles physically for covert
work and how to document audit results

o what kind of shop facilities are necessary

o how to approach administrative, civil, and criminal
enforcement actions

o how to publicize auditors' activities

o how to use the results to improve or manage the program.

Each is discussed below.

Maintaining Cover

Maintaining cover is critical for the effective operation of the
covert audit. If cover 1s compromised, at the very least station per-
sonnel may play along with the situation and attempt to perform in
accordance with what they believe the requirements of the program are.
In this situation, the covert audit may in fact be collecting informa-
tion which does not reflect ongoing practices in the program.

Many programs, including several with long experience with covert
investigations for safety programs, emphasize how stations frequently
ignore obvious tip-offs to an ongoing covert audit. Even when cars and
auditors are generally known, auditors are frequently able to perform an


The Station / Mechanic Evaluation report has been developed to assist
the Field offices 1m monitoring and enforcing the requirements of the
California Sig Check Program. By collecting and analyzing the TAS data the
report Gun be a valuable tool to provide a clearer picture of how a particular
station or mechanic 1s perforrung in the field.

The various areas Of informition on the report will be explained as
to their purpose ani the methods used to arrive at the results.

The report 1s detailed by station number and mechanic number.

Group 1:
Colum:

1. Total Tests
Lists the number of I (initial) or B (before repair) tests performed.

2. Prent Fail
Indicates the percentage of these I or B tests that failed.

3. % Dev from District Average Fail Rate
The percentage above or below the District average fail rate for I tests.

4. % Dev from District Average HC
The purpose of this field is to reveal chronic problems due to analyzers
with excessive drift or air bleeds during testing.
The district average HC for each standards category are determined and a
baseline is established. The average HC emission for each station and
mechanic is determined for each standards category. These averages are
divided by the district averages for each category that the sta/mec
performed tests in. This factor is summed and then divided by the number
of categories in which tests were performed. The result is multiplied by
100 to get the percentage deviation.

5. % Dev fron District Average CO
Same as number 4 for CO.

6. Total Tests
Lists the number of A (after repair) tests performed.

~

Prent Fail
Indicates the percentage of these A tests that failed.

8. % Dev from District Average Fail Rate (A Tests)
The percentage above or below the District average fail rate for A tests.

9. % Nev from District Average HC Reduction
Simular to number 4 above except this calculates the average emssion
reduction for HC. The first I test and the last A test for all vehicles
‘with multiple tests are used to determine the HC reduction. The figure
lasted on the report is listed as a positive or negative (-) vercent
deviation from the district average HC reduction.

19. 3 Dev from District Average CO Reduction
Sam: as nuuber 9 for CO.

-1-


LIST OF FIGURES

Structure of the Manual

Development of Enforcement-Oriented Covert
Auditing

Conceptual QA Framework for Enforcement-Oriented
Covert Audits

California Station/Mechanic Summary

Sample Tools for an Audit Set-Up Shop

iv

2-8

2-10


Group 3:

Colum:

Le

Percent Underhood Accuracy

The purpose of this field 1s to compare the underhood inspection infomation
specified in the emission control systems application manual to the actual
underhood data entered by the mechanic. The vehicle mke, year, number of
cylinders and engine size indicated on the test are compared to a table for
the corresponling vehicle make, year, number of cylinders aml engine size.
If the table and the actual underhood daca differ the mechanic will be
given an incorrect response. The number of incorrect responses are divided
by the total number of vehicles tested and a percentage of accuracy is
listed.

Erroneous Entries on Functional Check

This is used to see if the mechanics are performing the functional checks

in districts where the functional check 1s not required or are not performing
the functional checks in districts where they are required to do so.

When this is the case a counter is added to and the total of that counter

is listed on the report.

No. and % With Fast Repairs

This field is to indicate the percentage of time the A test begins in less
than 15 minutes following the completion of the corresponding I or B test.
The first number in this colum represents the number of times this occured.
The second number represents the percent of times this occured.

% A Test With No Repairs

This is used to see if mechanics are performing A tests without indicating
the repair action taken.

The A tests are checked in the repair action field. If no action was
indicated a counter is incremented. The total of this counter is divided
by the total number of A tests and that percentage is listed.

% Multiple Tests

This is used to see if multiple tests of the same type for the same vehicle
are being done. Multiple tests for a vehicle are checked to see if the

test type ( I or A) is the same as the Previous type. Where miltiple test of
the same type for the same vehicle are done the percent is calculated and
listed in the appropriate colum.

Number of Failed Tests With Certs

The purpose of this field is to determine if the mechanics are abusing

the emission reducton and exceeding cost limit to issue a certificate.

The number of times a certificate is issued due to a vehicle achieving

a reduction of emission and an exceed the cost amount is added to a counter.
This number can be converted to a percent and compared with the district
average.

3+


TABLE 3.2

POTENTIAL COMPARISONS TO BE USED IN CHARACTERIZING

I/M OPERATIONS*

Failure Repair Stations'
Rates Costs Warranties Approach
Analyzers by x
Manufacturer
Type of Service x x x x
Outlet
Vehicle Model- x x x
Year Groupings
Physical Location x x x

of Stations

* NOTE:

Warranty investigations will be oriented primarily towards

individual stations and possibly individual manufacturers.

3-4


available on the actual motivations of mechanics who purposely pass or
fail vehicles falsely. However, anecdotal information from current State
and local programs suggest that licensed stations do believe that their
return from an I/M program is maximized by test volume, and that test
volume is maximized by minimizing time associated with each test.

False Fails

False fails, either purposeful or accidental, are the subject of sub-
stantial concern.* When a station purposely fails a car which it should
pass, 1t is generally considered consumer fraud. A false fail can be
generated either by incorrect reporting of the result on the data form in
a program with manual analyzers, or by probing the exhaust pipe of a car
known to fail emission standards in a program with automated analyzers.
Deliberate disablements such as secretly disconnecting spark plug wires,
or misrepresenting the status of emission control components (for pro-
grams with tampering checks) are other ways to create false fails. It
should be pointed out that there is no data available to date which sug-
gests that false fails are a concern on a program-wide basis; rather it
only appears to be a problem on a station-by-station basis.

Based on historical experience with safety inspection programs,
some covert auditing programs have started by focusing on false fails.
Colorado, for example, initiated its program by investigating for false
fails, and found very few. The Colorado program has since changed its
emphasis and is now investigating false passes. As mentioned above,
reports from other programs also support the concept that false passes
are more of a problem in I/M programs than false fails. If it is deter-
mined that false fails are a real concern, then cars must be set up to pass
in order to investigate stations of concern.

Improper Data Recording

Improper data recording, while in and of itself is not a program
discrepancy, may prevent an accurate representation of program operations
from the data collected and analyzed. In programs with manual analyzers,
this problem is exacerbated by the fact that the inspector must accur-
ately read and record the data manually. Illegible and ambiguous markings
of inspection data forms may be discovered either through central office
review of records or through formal periodic field auditing. However,
some of the practices described above in this section overlap the area of
amproper data recording and may become confused with them. For instance,
1t may be important to know how much of a low failure rate 1s due to pre-
repairs versus false passes due to emission tests being omitted. The

* A false fail should not be confused with an error of commission, which
in a testing context means a vehicle which would pass an FTP but fails
a short test.


TI-2

TABLE 2.3

SUMMARY OF POTENTIAL EVIDENCE FOR SPECIFIC REPAIR PRACTICES

Passing/Falling Direct
Vehicles Observation
Program Practices Potential Evidence for Covert Audits Needed
Sticker/certificate High passing rates, fast Passing Yes
without a test, ! tests. Failing No
er without regard to
the results of a test.
Circumvention of rpm Discrepancies in rpm and Passing Yes
requirements. ! idle speed records.! Failing Sometimes2
Badly maintained probes.! Discrepancies 1n dilution records.3 Both Yes3
Leaving the hood closed.! High passing rates (tampering Passing Yes
inspections). Failing Sometimes2
Omitting fuel filler inlet High passing rates (fuel filler Passing Yes
or catalyst check.! inlet/catalyst checks). Failing Sometimes 2
Suggestions from station High failing rates, high repair Both No
personnel that repair work costs (possibly).
1s needed before a test.
Fraudulent fails (general). High failures, especially for Passing No

specific emission control components
and HC.



REASON FOR PRIORITIZING STATIONS FOR

TABLE 2.2

COVERT AUDIT BY STATION TYPE IN CALIFORNIA
(JULY 1984 - JUNE 1985)

New Car Used Car Independent Service Total
Dealers Dealers Shops Stations
Random Check 20 16 363 230 629
Complaints from
the Public 5 4 92 84 185
Suspiciously
hagh certi-
ficate issuance 0 0 7 4 1
Informant tips 2 5 34 28 69
Tips from QA
contractors
(formal auditors) e) 0 1 2 3
Referee actions (0) 1 5 1 7
Follow-up visits 2 10 110 115 237
Other 0 a 10 an 22
Total 29 37 622 475 1163



Station's Approach to I/M

Investigating in some detail what stations do in an I/M program
(how much diagnostics are performed, what a "typical" inspection 1s like)
may be one of the more difficult areas to investigate. However, to the
extent that the reasons for inadequacies in inspections or repairs are
determined, remedies (e.g., retraining of inspectors) are much easier to
develop.

Other Potential Areas of Investigation

I/M effectiveness studies may be difficult because FTP testing is
generally needed to demonstrate mass emission rate reductions. As an al-
ternative, idle emission reductions are frequently used as substitutes
for summary reports to legislatures, the public, etc. Special vehicles,
set-ups, and documentation are generally not needed. However, it is
helpful to have a breakdown of typical maladjustments so that the repair
aundustry may be tested with a set of vehicles which reflect the vehicle
population as a whole. Types of service outlets, vehicle model year
groupings, and vehicle makes are of particular interest in I/M effective-
ness studies, For 1981 and later vehicles, vehicle preparation is criti-
cal (see Chapter 5 and Appendix B).

Activities in Current Programs

Data collection and analysis studies are either in progress or
Planned in several areas. For example, the Salt Lake City/County
Health Department 1s considering a permanent group of stations for
data-gathering purposes. The Colorado Department of Revenue, in con-
junction with the Colorado Department of Health, is investigating
responsiveness to warranty claims by dealerships in the Denver metro-
politan area. An I/M effectiveness study is being performed by the
California Air Resources Board (CARB) in Southern California; utiliz-
ung the CARB laboratory in El Monte, California, full transient driving
cycle emission tests are being conducted.


TABLE 2.1

SUGGESTED CHECKLIST FOR DETERMINING OVERALL COMPLIANCE
WITH DECENTRALIZED I/M PROGRAM REGULATIONS

2.

3.

Are vehicles being tested according to established test procedures?

Are all tampering/fuel switching checks performed correctly?

If inspections are not being carried out completely or correctly,
what parts of the inspection pose the biggest problem?

Are results of the inspections recorded properly?

Are official tests being run only after adjustments are made?

Are the requirements for cost waivers being carried out correctly?

Are inspection certificates being issued properly?

2-2


Documented Vehicle File

Assemble a file for Field Representative

© = Original Pocument > = Copy of Docurent
Contents in file:

Repair Tacilaty Invoice fe)

Pepair Picility Test Report ie)

Certificute of Compliance (1f issued) 0

Job Order c
AFITR VEHICLE DATA Sheet oO
B.A.R. As Returned Printout fe]
PEFORE VEHICLE DATA Sheet fo

Asscmbly a file for shop records
Contents in file:
Job Order
B.A.R. "BEFORE" Printout
BEFORE VEHICLE DATA Sheet
Repair Facility Invoice

B.A.R. As Returned Printout

aQ fo} Qa a o oO

AFTER VEHICLE DATA Sheet


a centralized program. Simply stated, there are generally fewer stations
and inspectors who must be monitored 1n a centralized program; in addition,
inspectors ina centralized program, either by contract or employment, are
directly responsible to the 1mplementing agency.

I/M QA can be directed both at auditing proper inspections and at
auditing proper maintenance. Generally, I/M QA has been oriented towards
periodic visits to licensed stations, checking records, reviewing compli-
ance with administrative requirements, and auditing the performance of
emission analyzers, with a calibration gas of known concentration (USEPA,
1982; and Pienta, et al, 1985). Virtually all States with inspection pro-
grams have some sort of periodic surveillance. Program elements usually
aunclude not only periodic surveillance, but also referee stations for ve-
hicle owners who wish an independent check on test results (see for exam-
ple Wallauch, 1983).

There 1s a limit though to what structured, overt program elements can
accomplish (Carhart, 1985). Periodic overt field auditing can do much to
gather information which can indicate the direction of an I/M program; it
can also serve to bolster enforcement activities run by other parts of the
implementing agency. However, as indicated in Table 1-1, formal periodic
auditing cannot test inspectors' and mechanics' normal everyday operations
and practices directly. If, for instance, inspectors have the knowledge
and background to check for tampering of a certain emission control compo-
nent, they will almost certainly be able to demonstrate that capability to
an auditor if requested. However, in a routine workday, inspectors and
mechanics may not be as thorough as when they know they are being tested
by program auditors. They may even circumvent the requirements of the
program, such as passing cars that should be failing or passing vehicles
that are not inspected at all.

COMPREHENSIVE QA IN A DECENTRALIZED I/M PROGRAM

Discrepancies in decentralized I/M program operation have become more
evident 1n recent years (Cabaniss, 1984; and USGAO, 1985). In particular,
failure rates in several programs are well under expectations, especially
when compared to centralized programs with similar cutpoints and require-
ments. This phenomenon has developed despite the existence of periodic
field auditing. As a result, questions have been raised regarding the
ability of formal periodic auditing to assure that vehicles fail when they
should, the ability of stations to record program data properly, and the
ability of decentralized I/M as a whole to be effective in reducing emis-
sions.

Failure rates may be affected by many factors. Program phenomena
which may affect overall failure rates include "false fails" (vehicle
which fail but should pass), false passes (vehicles which pass, but
should fail) and "pre-repair" (vehicles which fail and are repaired but
are reported as initially passing). Program failure rates are only one
example of how a program parameter may need specific indepth investiga-
tion and analysis 1n order to reveal what is going on in an I/M program.
Auditing station operations enables program managers and others to obtain
information which would allow this and other kinds of program analysis.


TABLE 3.1

POTENTIAL AREAS OF DATA-GATHERING INVESTIGATION
FOR COVERT VEHICLES*

Subject

Sample Topics for Investigaton

I. Failure Rates

II. *Repair Cost

III. Warranties

IV. Stations' Approach to I/M

Are stations repairing or adjusting
vehicles and not reporting failures?
How are these actions affecting the
reported failure rate 1n the pro-
gram?

Are some stations charging a maxi-
mum amount (i.e., equivalent to

a cost waiver expense) for simple
adjustments?

Are simple adjustments being per-
formed for free?

Is there a difference in repair
cost in terms of the type of
service outlet?

How responsive are dealerships
to warranty claims?

Why do station personnel make
programmatic mistakes? Fraud?
Laziness? Ignorance?

*

I/M effectiveness and repair type investigations may be somewhat

more difficult to develop, based on the number of vehicles that
would be required to develop a good data base.

3-2


COLORADO DEPARTMENT OF PERSONNEL A7180A
A7182A

MOTOR VEHICLE EMISSIONS COMPLIANCE OFFICER

NATURE OF WORK

This is a multiple-range class describing the responsibility for conducting
field inspections, field compliance and quality review of licensed automobile
emissions stations and mechanics.

An employee in this class uses the required techniques to license emission
inspection stations, instruct emission mechanics and emission station
owners/managers in the proper emission inspection procedures for compliance
with the Automobile Inspection and Readjustment air statutes, rules and-
regulations.

Distinguishing Factors: Positions in the next higher level class are
- distinguished from positions in this class by having supervisory responsi-
bilities over employees in this class.

Range A

This range is characterized by the training/developmental nature of assign-
ments which utilize a wide range of technical skills and abilities and also
requires the application of specific knowledges of the statutes, rules and
regulations for the Automobile Emissions Program. Work includes considerable
public contact wherein judgment is exercised in executing departmental
policies. Initially, work is performed under close supervision, but as
training progresses, work is performed under general supervision.

Range 8

This is the full operating Emissions Compliance Officer level. At this level,
employees are expected to perform the full range of compliance, instructing
and enforcement duties for the Automobile Emissions Program. Emplayees'-

work is subject to only occasional review for technical accuracy, adequacy

of technical judgment and results. Employees receive instructions in very
general terms; detailed instructions in work process or procedure.are

rarely needed, except to deal with the most unusual or sensitive problems.

SOME EXAMPLES OF WORK

Performs compliance inspections of licensed motor vehicle emissions stations
and mechanics on a monthly basis. This requires the demonstrated technica)
skills in the evaluation and calibration of sophisticated electronic testina
equipment. An example is performing a gas span test on the Emission
Analyzer every 30 days.

Conducts field technical assistance and training as necessary to update
licensed emission mechanics and station owners/managers on changes in rules,
regulations and procedures.


MOTOR VEHICLE EMISSIONS COMPLIANCE OFFICER (Cont'd)
Page 3

Ability to interpret and apply laws and regulations uniformly.

Ability to maintain effective working relationships with other employees,
emissions station/dealer personne] and mechanics.

Ability to communicate effectively, both orally and in writing.
MINIMUM PREPARATION FOR WORK

Education and Experience

Three years of progressively responsible experience in automotive mechanics
which included motor vehicle tune ups and repair and diaqnosis of emissions
systems.

Substitution

Technical work in a motor vehicle emissions program will substitute for the
experience requirement on a year-for-year basis.

Full time teaching courses in tune up, emissions system diagnosis and repair
will substitute for the experience requirement on a year-for-year basis.

An Associate Degree from an accredited college with major coursework in auto
mechanics with the emphasis in tune up, emissions system diagnosis and repair
will substitute for two years of the required experience.

College, university or non-correspondence technical school course work in
auto mechanics including tune ups, emissions system diagnosis and repair
will substitute on a year-for-year basis for two years of the required
experience.

Necessary Special Requirement

Possession of a valid State of Colorado driver's license. A valid Air Program
Mechanic Certificate issued by the Health Department, State of Colorado must
be obtained prior to and/or during the probationary period.

NOTE

Positions are to be assigned to one of the ranges in this class based on
agency evaluation of the employee functioning level relative to the range”
descriptions identified in the class specification.

Proposed Published Date; 3/4/85
(LDS)


APPENDIX B-4

BACKGROUND INFORMATION OF STATION
PERFORMANCE REPORT
(FIGURE 2.3, CALIFORNIA BUREAU OF AUTOMOTIVE REPAIR)


THE ISSUE OF ENTRAPMENT

Based on surveys of operating programs in 1985, covert auditors
specifically do not ask inspectors or mechanics to perform work contrary
to program regulations. To do so would constitute entrapment,* i.e.,
inducing station personnel to commit a program violation. Instead,
auditors simply ask for an inspection.

Among the specific concerns voiced by State and local programs are
that:

© introducing a car that has altered or mssing emission control de-
vices, or a car known to fail emission standards, may constitute
entrapment;

© altering emission control devices on vehicles, even in the inter-
est of air quality improvement, may be a violation of Federal or
State law;

© driving a vehicle on public roads with altered emission control de-
vices may be a violation of State law.

A review of other states may be helpful to understand these and other en-
trapment issues. To be sure, entrapment 1s a matter of law, and if chal-
lenged, a State or local program might have to defend itself in court.
However, as a practical matter for the States that do use failing and al-
tered cars, entrapment has not been a major issue. Generally, presenting
a failing vehicle (even with altered emission control devices) has not
been considered entrapment, because stations are not being induced into
doing anything that is contrary to program regulations; all the station
has to do is to fail the vehicle. For state air pollution control laws,
obviously the State air pollution control agency should be consulted. It
should be noted that with respect to Federal laws and regulations, the U.S.
Environmental Protection Agency has allowed emission control modifications
for this purpose on a case-by-case basis.

PENALTIES, PUBLICITY, AND THE DURABILITY OF DETERRENCE

Figure 2-2 indicates that the "bottom line" of an enforcement-based
covert auditing program 1s deterrence and education. The reason for this
dual goal 1s that deterrence from detrimental program practices and educa-
tion towards implementation of the program as it is designed are both im-

* Black's Law Dictionary defines entrapment as “the act of officers or
agents of the government in inducing a person to commit a crime not
contemplated by him, for the purpose of instituting a criminal prose-
cution against him" (emphasis added).

2-14


DISCLAIMER

This summary report was furnished to the U.S. Environmental Protec
tion Agency by Engineering-Science, 10521 Rosehaven Street, Fairfax, Vir-
ginia, in fulfillment of Work Assignment 24, Contract No. 68-02-3888.
The opinions, findings, and conclusions expressed are those of the authors
and not necessarily those of the U.S. Environmental Protection Agency.
Similarly, mention of company or product names should not be considered
as an endorsement either by the U.S. Environmental Protection Agency or by

Engineering-Science.

vi


EXECUTIVE SUMMARY

The problems of quality assurance in a decentralized I/M program
are well-known. As a result, every current decentralized I/M program
does some formal periodic audit of licensed stations. However, formal
auditing cannot detect certain quality assurance problems. Specifical-
ly, while formal auditing can certainly test what station personnel may
know, it does not necessarily test what is actually done during inspec-
tions and repair.

Covert auditing allows I/M program managers to determine what is
actually happening in a program. "Undercover" cars have historically
been used 1n safety programs to gather evidence on stations which may
have been flagrantly violating the rules of the program. To be sure,
covert auditing may serve this role for I/M as well, but it also serves
to encourage comprehensive inspections and proper repairs. In addition,
at may be used to gather data on program operations.

It as the purpose of this manual to present program approaches
and techniques for covert auditing. Issues in both enforcement-related
and data-related covert auditing are described. Results of existing
covert auditing programs are discussed. Finally, some of the practical
issues in program development are investigated, and two model programs
(an initial program and a mature program) are presented.


FIGURE 2.1

DEVELOPMENT OF ENFORCEMENT-ORIENTED COVERT AUDITING

GOALS OF THE I/M PROGRAM

© Identifying vehicles with high emissions
(i-e., proper implementation of test
procedures)

o Repairing vehicles to meet emission cut-
points or other specific requirements
(i.e., proper implementation of repair
requirements)

PROGRAM DISCREPANCIES

Omission of test

Incomplete test

False passes

False fails

Improper data recording which would impair
confirmation of program results

oooo°0

INDICATORS OF PROGRAM DISCREPANCIES **

Sticker 1ssuance without test

Circumvention of rpm requirements

Leaving the hood closed

Badly maintained probes

Omitting fuel filler inlet or catalyst check

Suggestions from station personnel that

repair work is needed before a test

High waiver rates/incompletely documented

repair orders

© Arbitrary pass/fail decisions by station
personnel (manual analyzers)

o Arbitrary data recording on manual forms

o Inserting probe into exhaust pipe of vehicle
other than test vehicle

o Lack of certificate control

oo0o0o0o0

°

** NOTE - Depends on type of program.

2-3


CHAPTER 1

THE ROLE OF UNMARKED VEHICLES IN DECENTRALIZED
I/M PROGRAMS

INTRODUCTION

Decentralized motor vehicle emissions inspection/maintenance I/M pro-
grams* have long been favored by many State and local governments because
of the convenience they offer to motor vehicle owners. In addition, in
places where decentralized safety programs were already in place, legisla-
tures and agencies establishing I/M programs generally implemented decen-
tralized I/M programs to avoid any inconvenience and administrative dupli-
cation which would possibly result from parallel but separate inspection
systems. Despite the obvious advantages of a decentralized system, at the
same time the quality assurance (QA) problems inherent in decentralized
programs have always posed a challenge to program managers.

QA 1n a decentralized program involves four distinct steps:

© specification of the data and other information
to be collected;

o design and development of a surveillance system to deter-
mine whether procedural regulations are met;

© collection, reporting, and analysis of information; and

© modifying the system and/or the program on the
basis of analysis of results.

The key to developing an effective I/M QA approach, therefore, is to
identify what to monitor and what to control. Early assessments of QA
ussues 1n I/M tended to focus on the ability of individual mechanics to
perform proper repairs (USEPA, 1974). The need for oversight of licensed
stations became more clear as decentralized I/M was analyzed as a specific
program option (USEPA, 1978a and USEPA, 1978b). The essence of quality
control is to take actions which ensure that program implementation is as
consistent as possible with program design. In the case of I/M, it is more
difficult to match program design with a decentralized program than with

* For the purposes of this document, a decentralized I/M program is
any program which, at least in part, licenses privately owned
stations to inspect vehicles with respect to certain mandatory
emissions-related requirements, and to issue official certificates
or stickers as a proof of complying with those requirements.


Because of the increasing computerization of I/M data, with and
without computerized analyzers, station performance 1s much more subject
to scrutiny than it was in the late 1970's. However, few, 1f any, State
and local agencies actively use computerized summaries currently to
Prioritize their undercover activities. One state, which is just start-
ing an effort in this direction, is the State of California. A copy of
the California station performance summary report is provided as Figure
2-3. Note that this form 1s only a sample and that many other comparisons
may also be generated from similar data bases. However, several categ-
ories will depend on data generated and recorded by an automated analyzer.
For example, "No. and % with fast repairs," depends on an analyzer with a
clock or timer, so that "fast repairs" may be defined for a screening of
a computerized data base. (See Appendix B for a full explanation of the
California summary report).

As discussed above, other ways of prioritizing stations include
methods that have traditionally been used by safety programs with probable
cause covert auditing, including:

o complaints by vehicle owners;

o tips from QA auditors;

o reputation of a shop's activities from
other sources (police, etc.).

When combined with computerized data results, a list of target stations
can rapidly be developed. Each State or local program should have its
own criterla to determine what stations should receive priority for
covert visits. Among the questions that could be asked are:

How serious are the violations to be investigated?

How prevalent do these violations appear to be?

How do the violations to be investigated obstruct the
Proper implementation of the program?

How will covert visits to these stations help to deter
detrimental practices both at this station and the
program as a whole?

The last question is addressed below under “Penalties, Publicity, and the
Durability of Deterrence."

The question of how to prioritize stations then revolves around what
program practices need investigation. As shown 1n Table 2-3, observation
of the inspection 1s frequently not necessary if all that must be estab-
lished is that a station 1s passing cars that should fail. Direct obser-
vation or the equivalent 1s usually necessary to document what specific
practice 1s occurring (see Table 2-3, Note 2); however, in many instances
introducing a failing car can still allow the identification of a false
pass, though the exact reason may not be identified.

Therefore, depending on what program practices are of interest, cer-
tain data can be screened in computerized data bases. This information
then can be used in addition to information from the more traditional,
informal channels, (such as tips from the QA auditors), to prioritize
stations.


GUIDANCE MANUAL

USING UNMARKED VEHICLES IN
DECENTRALIZED I/M PROGRAMS

MARCH 1987

SUBMITTED TO:

EMISSION CONTROL

TECHNOLOGY DIVISION
U.S. ENVIRONMENTAL PROTECTION AGENCY

2565 PLYMOUTH ROAD
ANN ARBOR, MI. 48105

ES eNGINEERING-SCIENCE


TWO MODEL PROGRAMS

Table 5-2 presents an outline of two model programs. The first,
Alternative 1, is for a program which is just starting or modifying its
covert auditing program. It assumes very limited resources, and is
oriented towards integrating covert auditing into an existing auditing
Program. The second, Alternative 2, 1s for a more mature auditing pro-
gram, and may be used as a model either by an Alternative 1 program, or
by other auditing programs with a degree of existing covert experience.
These two alternative programs are directed towards attaining the basic
goals of a covert auditing program in a cost effective manner.


However, as explained above, formal periodic field auditing can go only
so far in identifying discrepancies 1n program operation. There is,
therefore, a need to obtain information in a way which will not affect the
everyday operations that the auditor may wish to observe. This need can
be met by covert auditing.

COVERT AUDITING - THE ROLE OF UNMARKED VEHICLES

In any decentralized inspection system, there 1s always a need for at
least a "last resort" mechanism to remove "problem" stations from the sys-
tem. Because administrative requirements generally require a reason for
license suspension or revocation, explicit evidence of violation of pro-
gram requirements is usually needed. Frequently the only way to obtain
first hand evidence 1s for the implementing agency to obtain that informa-
tion through an undercover vehicle.

Covert auditing is a generic designation for information-gathering
activities in an I/M program when the auditors do not identify themselves
as anything other than ordinary citizens. Generally the term "covert au-
diting" has been used in the same way as the term "undercover vehicles,"
with the implication that such vehicles are engaged only in uncovering
consumer fraud and enforcing program regulations. To be sure, unmarked
vehicles can and have been used for such purposes, not only for decen-
tralized I/M programs, but also historically for safety programs and con-
sumer protection investigations. It 1s a mistake though to believe that
unmarked vehicles may only serve to support enforcement. As indicated
above in the previous section, the essential purpose of covert auditing
1s to find out what is happening 1n an I/M program, while identifying
where program implementation is failing to track program design. Covert
auditing programs for decentralized I/M programs should, therefore, be
developed with a clear goal of what information is to be gathered, and
how the anformation will be used beyond just enforcement of program regu-
lations. There are many ways this information can be used to improve
program performance. Details on specific data to be gathered and how to
gather it are presented in Chapters 2 - 3.

OUTLINE OF A COVERT AUDITING PROGRAM: AN INTRODUCTION TO PROGRAM PLANNING

Once the specific data to be gathered 1s established, the basic struc-—
ture of the audits themselves must be evaluated. Two basic issues which
need to be decided early are whether the covert vehicles themselves should
be set to pass or fail, and how to establish incontrovertibly the results
of the investigations.

Setting Up Vehicles to Fail or Pass

Historically there has been concern that decentralized inspection
systems, though they offer substantial convenience to the vehicle owner,
also open the way to consumer fraud. Specifically, there 1s an inherent
conflict-of-interest in a decentralized inspection program, where a sta-
tion stands to gain financially from failing a vehicle which is being
tested; as a result, a station mechanic may have an obvious motivation to


TABLE 1.1

CAPABILITIES AND LIMITATIONS OF FORMAL PERIODIC
FIELD AUDITING IN DECENTRALIZED I/M PROGRAMS

Capabilities

Can review an inspector's ability to work with the analyzer
on-site.

Can determine knowledge of test procedures and record keeping
procedures.

Can analyze basic approach to check for emission control
components.

Can identify discrepancies in repairs if repair orders
are available.

Can verify accountability of inspection certificates.

Limitations

Cannot confirm the station's usual method of testing vehicles.
Cannot verify or disprove by itself fraudulent repair practices.
Cannot verify or disprove by itself inadequate knowledge or

implementation of program procedures by inspectors and
mechanics.

1-3


California

Tocunented Vehicle Procedures - Outline

‘lL. Receive job order from Pistrict Office.

2. Review job order and assign to shop technician.

3. Perform documentation to vehicle as per job order.
4. Run I/M test on T.A.S. and obtain 2 printouts.

5. Miter documentation performed ax! 1 TAS printout into
docunentation log book.

G. Complete a "BMFORE" VEHICLE CATA sheet on the vehicle.
7. Staple remaining T.A.S. printout to back of job orver.
8. Release vehicle to Field Representative.
a. See Attachment, Program Representative's Duties
9. Inter release information into mileage log (Date, Time, Mileage)

10. Receive (returned) vehicle, test report, invoice, and CofC (if
issued) .

11. Enter the information into mileage log. (Date, Time, Mileage
and PR's signature)

12. Reinspect and retest vehicle on T.A.S. to document its as returned
condition and obtain 2 printouts.

13. Complete a "AFTER" VEHICLE DATA sheet on the vehicle.

14. Finish campleting the as returned information on the job order.
15. Write Declaration if requested by Field Representative.

16. Canplete a Cost Shect if requested by Field Representative.

17. Complete Documented Run statistics.


TABLE 5.1

CHECKLIST OF QUESTIONS FOR ENSURING THAT COVER IS
MAINTAINED DURING A COVERT INVESTIGATION

If the auditor has to leave a phone number, is the number local?

If the vehicle must be left, 1s there someone at that phone number
to answer and maintain the cover of the investigation?

Is the combination of the auditor (including the way the auditor is
dressed) and the vehicle reasonable for the type of clientele and
business normally performed by the station under investigation?

Is the service the auditor asks for consistent with service requests
station expects?

5-3


CHAPTER 3

COLLECTION OF DATA ON PROGRAM OPERATIONS

Covert auditing presents opportunities to I/M program management to
collect information on what is happening in the program without affecting
anspectors' and mechanics' behavior. This advantage, though, is not and
should not be focused on enforcement activities alone. Information
gathering about the program in general can be enhanced by having people
ain the field watching what happens.

Similar to enforcement-oriented auditing, auditing for the purpose
of data collection must be well-planned from the beginning. It is impor-
tant to know what information to look for, plan a system to gather that
information, and analyze the information 1n a systematic way.

Among these areas which could be the subject of investigations are:

Failure rates

I/M effectiveness

Repair cost

Station's approach to I/M

Types of repairs
Warranties

oo00000

Table 3-1 presents sample topics which could be explored within some of
these areas.

COORDINATION WITH ENFORCEMENT-ORIENTED COVERT AUDITING

Not every program can do a special study, and setting up methods for
collecting special data on a routine basis often takes time. Collecting
data to characterize program operations may be a valuable first step in
setting up an enforcement-oriented covert auditing program.

As discussed in Chapter 2, the goals of an I/M program ultimately
can be compromised by detrimental program practices (e.g., fraudulent
fails, false passes, etc). With limited resources for covert auditing,
at is more cost effective to focus the potential deterrent effect of a
covert auditing program on the practices which hurt the effectiveness
and/or credibility of the program the most. To be able to determine what
practices to highlight, program management must know how prevalent dif-
ferent practices are.


TABLE 5.2

TWO MODEL PROGRAMS

Program Element

ALTERNATIVE 1

ALTERNATIVE 2

Prioritization
of Stations

Maintaining
Auditors’
Cover

Passing/Failing
Cars

Personnel

Team
Organization

Kinds of
Vehicles

Vehicle
Procurement

Work off of complaints and
the highest passing rates.

Start with formal auditors
working adjacent territo-—
ries; experiment with dri-
vers outside the program.

See comments below; use
both passing and failing
pre-1981 vehicles to start;
if only one vehicle may be
procured, use a failing
pre-1981 vehicle.

Use existing formal audi-
tors and start recruiting
outside people as possible.

Start with splitting the
time of formal auditors
between covert audits and
formal audits.

Emphasize pre-1981 vehicles
of a common make.

Obtain at least one program
vehicle but plan for tran-
sition to a different vehi-
cle.

Develop a computerized
data system which allows
screening of unusual
data.

Analyze situations which
may expose an audit; mi-
nimize visibility of par-
ticular vehicles and
auditors.

Develop audit vehicles
which are directed to-
wards detrimental pro-
gram practices found to
be prevalent in the pro-
gram; experiment with
1981 and later failures.

Use a mix of employees and
non-employees; determine an
optimum balance based on
costs, maintaining auditors’
cover, and flexibilty in
planning audits.

Maximize the number of peo-
ple which can be called on
to be auditors (office per-
sonnel, formal auditors,
etc.)

On the basis of data collec-
tion about program operations,
emphasize vehicles which are
involved in program discre-
pancies; use some 1981 and
later vehicles.

Prepare a list of all possi-
ble vehicle sources (inclu-

ding private vehicle owners);
try a variety of approaches.

5-8


° misunderstanding or ignorance of the program;
° partial or total disregard of the program;
° fraudulent implementation of the program requirements.

For station personnel who do not really know I/M program requirements,
education and retraining 1s essential. However, some state's experience
(e.g-, Colorado) indicates that continuing conferences for some stations
seems to perform no useful function; at this point, stations who continue
to perform inspections or repairs incorrectly must be considered as risks
for correct implementation of the program. Nevertheless, some stations
do manage to revise their procedures.

Several State and local programs interviewed mentioned that the
swiftness, sureness, and severity of penalties were the key to successful
deterrence from a covert auditing program. The quality of deterrence may
be debatable (see below), but qualitatively there seems to be some merits
to this argument. Swiftness of penalties may be aided by fining authority
("ticketing") by the regulatory agency. As of January 1986, California
was granted this authority, and determining the impact of this approach
may be possible.

The deterrence effect of publicity must also be considered. First,
however, who publicity will reach must be assessed. Among the method I/M
programs have tried are:

° trade association newsletters;
° press releases;
° courses through vocational educational institutions

and service station organizations.

All of these options are to be encouraged, but realistically programs must
ask first who will be reached by specific publicity. A trade association
newsletter, for instance, is always a meaningful approach of outreach to
the service industry, but in and of itself, it may not be sufficient.
The Regional Office of New York DMV, for instance, reports that only 25%
of licensed stations belong to a professional trade organization (e.g.,
the Long Island Service Station Association). In other words, outreach
and general publicity to maximize the impact of limited covert auditing
must be looked at as a total package.

2-15


APPENDIX B-3

AUDITOR POSITION DESCRIPTION


ACKNOWLEDGMENTS

Engineering-Science wishes to express its gratitude to Federal,
State, and local personnel who aided in the development of this manual.
In particular, Engineering-Science would like to thank Bob Grosso and Phil
Ryan of the New York Department of Motor Vehicles, John Duncan of the
Colorado Department of Revenue, and Gary Hunter of the California Bureau
of Automotive Repair for setting up and coordinating two-day program
visits in Westbury (Long Island), Denver, and Sacramento. Finally, Engi-
neering-Science would like to express its appreciation to Phil Lorang,
Chief of USEPA/ECTD's Technical Support Staff, Jane Armstrong, USEPA's
Project Officer, and John Cabaniss, USEPA's Project Coordinator, for their
assistance, guidance, and cooperation.

vil


first is a data recording problem and does not necessarily indicate that
program effectiveness 1s being compromised. The second is a program abuse.

PRIORITIZING STATIONS: AN AID TOWARDS DOCUMENTATION AND DETERRENCE

There are many ways of prioritizing stations, but they can generally
be classified into periodic random auditing, probable cause auditing, and
“problem station" auditing. For the purposes of this document, these
classifications can be defined as follows:

_Periodic/Random Auditing - Auditing with a formal or informal
schedule to visit all stations in program within a _ specified
tame period (i.e., strictly speaking, no prioritization).

Probable Cause Auditing - Auditing which targets particular
stations on the basis of the frequency of complaints, discre-
pancies on data records, etc.

"Problem Station" Auditing - Auditing which uses covert investi-
gations only to obtain evidence against a station for the pur-
poses of suspending or revoking that station's license.

Depending on the program, these systems could be combined into a hybrid
approach.

Currently, there are programs that utilize each of these approaches.
Colorado utilizes a combination of periodic auditing and probable cause
auditing. The Department of Revenue, the administering agency in Colorado,
visits each licensed station once a year. They also keep an informal list
of "target" stations, which are visited as convenient. New York is a good
example of a State which utilizes probable cause to target stations.
Though the Regional DMV offices do receive computer printouts of station
records, complaints and tips from periodic inspectors are the main fac-
tors used to prioritize stations for covert visits. As a result, the
State visits from 10-25% of the stations per year. In some ways, "“prob-
lem station" audits really are a subset of "probable cause" audits. Ina
program with a “problem station" approach, fewer audits are conducted,
because the primary reason for performing an audit is to gather evidence
agalnst a station which 1s almost certainly committing serious violations
of the program regulations. As an example of how one State decides to
perform covert visits, a summary of reasons for covert audits in Califor-
nila 1s presented in Table 2-2.

Ultimately, the choice of an auditing prioritization scheme depends
on the goals of a covert auditing program, 1.e., documenting and deterring
the practices which lead to program discrepancies (Figure 2-1). To be
sure, 1f stations know that covert auditing is not conducted except in
rare cases, stations' standard operating practices for I/M will probably
not change much. While periodic covert visits to every station may not
be possible due to limited agency funding, having covert audits on a
regular basis clearly increases the chance for documenting and deterring
detrimental program practices. Therefore, the need for efficient priori-
tization of stations is clear, and prioritization, even with relatively
few visits, can still lead to deterrence (see Figure 2-2).

2-6


State of New York - Department of Motor Vehicles Case No
INVESTIGATION REPORT
Facility No
(J conceaceo IDENTITY (J compcaint SANS (HO Ue ONI
-asé Print or Type All Entries and Use Black Ink
Name of Complainant Name of Facility
lress - Number and Street D/v/a (if applicable)
State Zip Code Address - Number and Street
1 eiephone No (including area code) City State Zip Coge
tcle Year, Make, Model cy! Telephone No (including area code) | Facility Expiration Oate
WTS Plate No. ‘Name and Ident No of Certified Inspector
@ of Transaction Odometer Reading at Time of Date of C 1 /Complaint Odometer Reading at Time of
Transaction Investigation Investigation
_fsuant to N. Y Vehicle and Traffic Law, Section: (Central Office Use Only)
Insp Cert Repair/Body Other Junk &
Recommendation Station Inspector Sop Dealer Disman ler Salvage Facilities*
Dino Action QO Q i) oO oO i)
NOI O a) Oo im) Q Oo
_ Osusp Pend Hrg ia] Q oO o o Oo
Oueanng 0 Oo Oo a) oO o
Gi Inspection Waiver Oo Oo
Ootner 0 0 ia) QO o i)
“Other Junk and Salvage Facilities’, specify here
(DJ unregistered Repan Shop (st Complaint Case No (s) (Refund to Consumer $.

7] Restitution Case (J Rework a1 No Charge

7 Special Instruction(s) a C1 insurance Claim No

. J subpoena

5 Hearing Notice to

pector’s Signature Date AFI/BRI No

dervisory Signature Date Region

VS-35 1 (7/84)



he

Wear appropriate attire (work, hunting, or fishing clothes) to
assume the identity of an ordinary motorist.

If possible, carry lunch box, local newspaper, packages,
fishing rods, tools, etc. in vehicle to strengthen impression
that you are an ordinary motorist.

Tune radio to local station.

When necessary, telephone inspection station in an attempt to
have the vehicle inspected. This will ensure that you do not
waste too much time waiting around,

vieit the inspection station in an attempt to have the vehicle
inspected. If the inspection station says they are busy, ask
for an appointment.

2, Inspection

NOTE: When presenting the vehicle for inspection, be careful not

to say anyting that might be considexged entrapment. A
single statement such as "I need a state inspection", or
"Can I please get my car inspected" is all that is
necessary.

Keep the vehicle under observation at all times.
Some suggestions:

1, Strike up conversation with the inspector, express curiosity
about the inspection, etc.

2. If the inspector asks, participate in the inspection by
applying the brakes, operating lights and directional signals

3. Ask to check exhaust system for leaks, etc. when the vehicle
is jacked up, or tell the inapector you're getting a rattle ‘fro
the tail pipe.

Do not, under any circumstances, attempt to hurry the inspectior.

If the inspector advises that adjustments or repairs are
necessary, use the following guide in making a decision on
whether to have them made at the station,

1. If the concealed identity inspection was conducted because
a routine inspection revealed many adjustments on a certain
item, or as a result of complaints concerning unnecessary
repairs, adjustments can be expected, In these cases the
repairs or adjustments should be made,

2, %If the inspection was completed in a satisfactory manner,
adjustments should not be made (except in cases of
unnecessary repairs, as indicated in item 1 above).

3. %I€ repairs are recommended by’ the station,, do not authorize
them, except in situations where the Regional Director has


In Colorado at least, 1t appears that inspectors are less likely to ig-
nore an exceedance of the I/M cutpoints than they are a missing or modi-
fied fuel filler inlet restrictor or catalyst.

As discussed above, there 1s very little data available which indi-
cates over time what covert auditing approaches have worked in terms of
deterring detrimental program practices. Therefore, it is helpful to
characterize program operation so that a covert auditing approach can
either be developed or modified to focus on specific areas of a program.
Development of a data baseline may be a useful first step towards this
goal.

DETERRENCE THROUGH PUBLICITY

An example of what can be done in terms of publicity for covert
auditing activities occurred in California in the summer of 1985. The
California Bureau of Automotive Repair (BAR) had been receiving complaints
from buyers of used cars that cars were being sold with altered or missing
emission control devices. Because California runs a periodic I/M program,
which includes includes numerous tampering checks, purchasers of vehicles
falsely certified for sale would then be faced with paying for modifica-
tions of the vehicles to be able to pass I/M requirements. To substan-
tiate the complaints, BAR bought several sample vehicles which had missing
or altered components and removed the vehicles to the BAR shop to verify
and document their condition. Then, in a coordinated media approach, with
the aid of the California Highway Patrol, the car lots which sold the cars
were cited for violation of specific state laws and regulations. There
was substantial press coverage, not only in Sacramento, but in other
parts of the State as well. Informal feedback indicated that prospective
vehicle purchasers had heard of the state's operation on the car lots,
and were now asking questions of the car lots before buying a vehicle. The
owners of the car lots now appear to be aware of the fact that the state
could still occasionally buy a car off a lot.

Of course, this example presents one incident that could affect one
portion of an I/M program. There 1s no indication of how long the deter-
rence generated by this operation will last, no matter how good it was
initially. However, this incident does underscore the concept that publi-
city about ongolng operations is a useful part of an overall strategy to
encourage both deterrence and education. It also raises the possibility
of using consumer awareness to help in that process.

DEVELOPMENT AND DISPOSITION OF CASES

As described above, California does prioritize 1ts audits and as a
result does try to target stations which are most likely to be performing
inadequate inspections or repairs. Table 4.2 presents a summary of over
1400 audits performed in California. As presented in the table, a sub-
stantial portion of the audits (61.3%) resulted in a documented violation


TABLE 4.2

ADMINISTRATIVE SUMMARY
COVERT AUDITS IN CALIFORNIA*

Results of the Investigation (Total = 1404)
#
No violations 412
Aborted 87
Violation 861
Other (Please specify) 44
Disposition of Case (Total = 1317)
Office conference 390
Submitted for disciplinary action 119
Submitted for administrative action 58
NOV issued 283
Closed no action 445
Pending 22
Type of Station (Total - 1173)
Dealer (new) 29
Dealer (used) 37
Independent. 628
Service station 479

* Approximately 50% random and 50% targeted audits.

4-4


2.

3.

APPENDIX B

BACKGROUND MATERIAL FROM
ONGOING I/M PROGRAMS

Covert vehicle investigation report forms
Covert vehicle investigation procedures
Auditor position descriptions

Background information of station performance

report (Figure 2-3, California Bureau of Auto-
mative Repair)


TABLE 2.3 (continued)

SUMMARY OF POTENTIAL EVIDENCE FOR SPECIFIC REPAIR PRACTICES

e@T-2

Passing/Fa1ling Direct
Vehicles for Observation
Program Practices Potential Evidence Covert Audits Needed
Inaccurate waiver High waiver rates; short time Failing No
processing. periods betweeen initial and
second tests.4
Arbitrary pass/fail High passing or high failing Both No>
decisions. rates.
Arbitrary data recording High passing or high failing Both No5
on manual forms. rates.
Inserting exhaust pipe Small standard deviation of Failing Yes
anto vehicle other than test scores.
a test vehicle.!
Waivers to 1981 and later High waiver rates for 1981 and Falling No

vehicles which are serviced
by untrained personnel.

later vehicles; short time periods
between initial and second tests;
high incidence of specific, un-
anticipated repairs for late model
vehicles, carburetor repairs on
1981 and later vehicles and fuel-
injected vehicles.



MOTOR VEHICLE DIVISION
"AIR" PROGRAM
COVERT VEHICLE EMISSIONS CERTIFICATION

Date: Time Investigator

Vehicle Make: Year License VIN Number

VISUAL INSPECTION

Air Injection System Catalytic Converter Fuel Restrictor

EMISSIONS STANDARDS

Vehicle CO Standards Vehicle HC Standards

$ ppm

EMISSIONS READINGS

CO &. Idle HC PPM
2500

“ t CERTIFY THAT | HAVE INSPECTED THE ABOVE-DESCRIBED VEHICLE IN ACCORDANCE

WITH THE RULES AND REGULATIONS GOVERNING COLORADO AUTOMOBILE INSPECTION AND
READJUSTMENT PROGRAM. THE ABOVE-LISTED EMISSIONS READINGS ARE CORRECT FOR

THE VEHICLE, OR THE VEHICLE FAILS THE VISUAL INSPECTION OF EMISSIONS
EQUIPMENT AS NOTED, AT THE TIME AND DATE LISTED.

Signature Title Date

Subscribed and sworn to before me this day of 19

My Commission Expires:

Notary Public


given pre-approval. [In all other cases, request a written
estimate on the station's invoice.

After paying for services, obtain a receipt for the inspection

, and any repairs made. If no invoice is given, attach a signed

statement to the report indicating the amount paid.

NOTE; .If the AFI ia told to wait {n an area where he
will be unable to properly observe the inspection,
he should wait until the vehicle is brought in and
then identify himself and request that the inspection
be performed in his presence. This eliminates the
waste of time and money associated with an unobserved
inapection.

3, Post-Inspection

a.

b.

Evaluate inspection and complete Inspection Checklist
(Form VS~83.1) as soon as possible,

Remove inspeetion aticker from vehicle agd cover adhesive
side with a transparent material.

If it is the policy of the Regional Office to return
to the station shortly after the CI has been completed:

@ Show identification and review inspection with inspector
and/or owner.

@ For safety inspections only, obtain VS-1074 as proof of
CI and give atation a receipt for it.

e@ For a safety/emissions inspection only, {if no
analyzer receipt was obtained after the inspection,
inform your Senior AFI who will aasign an AFI to
perform a program audit and obtain the receipt.

@ Perform program audit on records and equipment.

Always complete the Investigation Report (Form VS-35,1).
Attach your written report, removed inspection sticker,
invoice, and any other pertinent information to the VS=35.1.

If€ an improper inspection hs been completed, complete CI
Inspection Notification Letter (Form VS-89) in triplicate,
Mail one copy to the station, one copy to the certified
inspector, and attache one copy to your completed report.

Disposition of Materials:

1. Forward completed report and attachments to Sr. AFI.

2. Forward copy of VS-35,1 and original receipt or statement of
payment to person responsible for replacement of @I funds.


LIST OF TABLES
Capabilities and Limitation of Formal Periodic
Field Auditing in Decentralized I/M Program

Interrelationships of Program Elements 1n Alterna-
tive Approaches to Covert Auditing

Suggested Checklist for Determining Overall Com-
pliance with Decentralized I/M Program Regula-
tions

Reason for Prioritizing Stations for Covert Audit
By Station Type in California (July 1984 -
June 1985)

Summary of Potential Evidence for Specific Repair
Practices

Potential Areas of Data-Gathering Investigation
For Covert Vehicles

Potential Comparisons To Be Used in Characterizing
I/M Operations

Summary of Covert Investigations in the Colorado
I/M Program (January 1 - November 22, 1985)

Administrative Summary-Covert Audits in California

Checklist of Questions for Ensuring That Cover is
Maintained During A Covert Investigation

Two Model Programs

z2i

1-6

2-11

3-4


Enforcement

Enforcement may take many forms. The typical hierarchy in current
inspection programs include:

warning letters

office conferences

administrative hearings

license suspension and/or revocation

ooo°o

Criminal actions sometimes are not pursued by programs because they be-
lieve that enforcement purposes are frequently served adequately by the
results of administrative actions. Sometimes actions against stations may
not even reach the administrative hearing stage. New York, for instance,
utilizes a plea bargaining approach which substantially reduces their
hearing load. Some programs are able to suspend stations without hear-
ings, but many programs are bound by administrative procedure statutes to
follow a specified legal protocol before a station can be removed from
the system.

With limited resources, covert audit programs must avo1d resource
intensive, time consuming actions whenever possible. Hearings and even
criminal action will occasionally be required, but they should be kept
to a minimum. To the extent that enforcement and, for that matter, re-
training can be accomplished by letters and office conferences, resources
will be saved for other activities.

However, any plan to minimize case or hearing loads need to be re-
viewed carefully prior to implementation. On the basis of observations
and experience of existing programs, swift and meaningful penalties are
necessary. Fines may be useful if they are large enough to deter program
abuses but small enough to ensure that they are administered and paid
routinely.

Publicity

Part of deterrence for station personnel is the knowledge that they
could be inspecting a covert vehicle at any time. Stations have contacts
with three main groups and organizations as a result of an I/M program.

o implementing agency personnel
o other stations and their trade associations
o the public

A review of experience with current auditing programs indicates that
having each of these groups involved is helpful. The implementing agency
can stress during training, certification, and formal audits that covert
auditors are actively checking stations. Trade associations can be enlis-
ted to work for the credibility and reputation of the program and their
members. The public may be educated by the mass media on specific abuses
that they can look for during inspections and repairs.

5-6


There are of course many ways that State and local governments can
organize their teams, and there are many constraints to work with. One
interesting approach to this issue has been in California where the au-
ditors themselves have minimal responsibilities, and basically only in-
troduce the vehicle into the target station with a plausible story. The
pre- and post-audit documentation by the shape of the vehicle allows the
verification of what happened during the audit. As a result, the audit-
ors themselves may be anyone who is able to maintain a reasonable cover.
California uses former State employees, other retirees, local law stu-
dents, and a variety of other people on a pay-as-you-go basis. This ap-
proach avoids having to hire new employees.

Other programs have initiated similar approaches; employees of the
umplementing agency or sister agencies, have been used with similar suc-
cess. Working with a limited number of employee-auditors can work also,
but then program management needs to be more aware of potential factors
which could compromise the auditors' cover. A group of diverse auditors,
plus a group of diverse vehicles (see below) eliminates this problem.

Organization of Personnel

The two basic approaches used by programs to date (other than the
California approach described above) are to:

o use dedicated covert auditors; or
o split the time of auditors between covert and formal
auditing.

There 1s no consensus on the better approach. Dedicated covert auditors
may have a very well-developed sense of their program responsibilities,
which may aid them in developing a commitment to their task. Splitting
the time of auditors on the other hand, may give auditors a better sense
of the program as a whole, and gives more variety to an auditor's day.

Colorado uses a team approach where one one auditor does covert
audits in another auditor's formal auditing territory and vice versa;
this approach may allow for trading of information about particular
stations.

Vehicles

The variety of vehicles, like the variety of auditors, is a plus for
the program because 1t helps to maintain a program's cover. Programs have
used many different types of vehicles. Vehicles have been borrowed from
employees, and purchased outright. New York has had success in procuring
vehicles from police impound lots.

The type of vehicles chosen will depend on the sort of audits planned.
For example, it is difficult to try to introduce an inexpensive compact au-
dit car into a luxury car dealership! At the very least, unexpected sta-
tion/audit car combinations can make station personnel wonder what is hap-
pening. Borrowing cars on an as-needed basis, and having a designated
fund to pay for their use, is one method around this problem.


MOTOR VEHICLE EMISSIONS COMPLIANCE OFFICER (Cont'd)
Page 2

Examines station records to insure report forms are accounted for and properl)
filed; examines New Car Logs at dealerships for proper issuance on Emissions
Certificates and their accountability.

Does necessary follow-up due to supervision of emission mechanic/station
licenses or reinstatements.

Resolves consumer complaints concerning emissions related issues.

Prepares monthly contact forms for stations contacted; prepares investigative
reports on violation of the "AIR" program.

Periodically conducts covert investigations on licensed emissions stations
and mechanics.

Attends and testifies in the informal hearings conducted by a Motor Vehicle
Emissions Compliance Supervisor and/or the formal hearings held in the Motor
Vehicle Hearings Section.

Performs related work as assigned or required.

KNOWLEDGES, SKILLS ANDO ABILITIES

Thorough knowledge of basic tune up procedures, and ability to use tune up
equipment.

Considerable knowledge of "AIR" program statues, rules and regulations.

Considerable knowledge of relevant state and department rules, regulations,
policies and procedures.

Considerable knowledge of the functions involved in the control of harmful
pollutants from internal combustion engines.

Considerable knowledge of ignition systems, fuel systems, emission systems
and internal combustion engines.

Knowledge of the principles underlying the operation, use, calibration and
maintenance of scientific electronic instruments used in the testing of
motor vehicle emissions.

Kriowledge of the purpose, function and effect of emission related parts of
an automobile.

Ability to perform all parts of an emission inspection and test.
Ability to use a gas span, use and calibrate an infrared analyzer.

Ability to read, comprehend, interpret and translate highly technical
emission data.


STRUCTURE OF THIS MANUAL

The purpose of this chapter has been to describe QA in decentralized
I/M programs, and the conceptual role of unmarked vehicles in enhancing
I/M QA. Chapters 2 and 3 describes specific issues involving enforcement
oriented and data collection covert auditing programs. Chapter 4 presents
results available from existing programs. Chapter 5 then discusses pro-
grammatic issues for setting up or expanding covert auditing programs plus
suggestions for two specific model programs. Appendices are provided in
the areas of shop guidance and background materials from State programs.
Figure 1.1 presents the overall structure of the manual.

1-7


OR 1362 (3/84)
INVESTIGATOR / AGE __ 42-16-410

COVERT VEHICLE REPORT

DATE. TIME.

AIR PROGRAM REPORT NO.

INSPECTION OBSERVED yes [J] wo CJ STICKER NO.

MECHANIC NO.

STATION SIGN POSTED ves (1) no CI

EMISSIONS STANDARDS VISIBLE yes [J sno (1 tn customer area []) snoop (]
HOW WAS VEHICLE YEAR VERIFIED REGISTRATION [7] vin (1) emission vecaL Corner C]
STATION STARTED INSPECTION FORM yes ( wo OJ

CHECKED EMISSIONS LABEL FOR DEvIcES yes [_] no CJ nja CI

INSPECT CATALYTIC CONVERTER ves CI no () wa CI

INSPECT AIR SYSTEM yes (J wo wa CI

INSPECT FUEL RESTRICTOR yes [1 wo CJ wa CT

WHAT VISUAL ITEM MUST BE REPAIRED/REPLACED catatyst [] air [J restrictor [1
MAXIMUM TIME ALLOCATED BY STATION FOR REPAIR sovays CJ] omer CI

HOW WAS ENGINE TEMPERATURE VERIFIED cuace [J radiator [J] customer [1

ZERO & ELECTRICALLY SPAN ANALYZER ves CJ no CI

PROBE INSERTION/ADAPTER USAGE yes [CJ no CJ

ACCELERATE ENGINE TO DEACTIVATE OVERRIDE DEVICES (1980 & OLDER) yes [_] no C) wa CD

» WETER READINGS 1st HC co.
2500 He co.
2ND He co.

WAS TWO SPEED TEST PERFORMED ¥;//TACH. YES [_] no CC] w/a CI

WAS DRIVER INFORMED OF FIRST EMISSIONS TEST FAILURE YES [_] nso 1] wa C1

“VERBALLY INFORMED OF 207(B) WARRANTY STATUS ves CJ nso C) w/a CO
4OW WERE ENGINE SPECIFICATIONS DETERMINED poo; CJ] vecatr []~ orner
TACHOMETER HOOKED UP ves OO no C] n/a ()
?ROPANE ENRICHMENT USED ves (J no C1] wa 1

SARBURETOR ADJUSTMENT METHOD toco CI LEAN Best [1] PROPANE ENRICHMENT [_] oTHER [_

AFFIXED EMISSIONS CERTIFICATE apsust [1] pass [[] sticker CERTIFICATE NO
AFFIXED TEMPORARY CERTIFICATE yes CT no OM wa 1
SORRECT STANDARDS USED ves ( wo (1

COMMENTS /RECOMMENDATI ONS



BUREAU OF AUTOMOTIVE REPAIR

DOCUMENTED VEHICLE JOB ORDER

VEHICLE

YEAR

MILEAGE IN

MILEAGE OUT

DATE IN
OATE OUT

SOURCE

ORandon
O Fetlow-Up
O Referee

O Comptaint
O Cert Sate
OQA

Ol

17

19

Other

RESULTS

ONegative
O Positive

RELATED
JOB ORDER

oe at

1, Detail Task Code Is inserted in the first column under Detail Task and Operation Code heoding with the Operation Code beginning

In the third column.

2. If more space 1 needed for comments use a number 1. 2. etc. under comments and use the some number on reverie side under

remarks and follow with comment.

DETAIL TASK
AND

OPER. CODE

COMMENTS

TECHNICIAN

TIME

CASE NO.

FIELD

REPRESENTATIVE

PHONE

NOTIFIED

TIME
NEEDED

RELIEF
NUMBER

REPAIRS

IMMEDIATE

DELAYED

DOCUMENTATION

IMMEDIATE

OELAYED

SERVICE

EMPLOYEE | ACT. TIME
NUMBER - Yo

LUBRICATION

OlL CHANGE
i

our ceas sor

Olt FILTER

AIR FILTER

WASH RACK

ANTI FREEZE

LABOR

MATERIAL

CHECKED
eRaKes

cwIcH GQ HORN QQ)
ucHTS GQ wirerss

TURES a

‘COMPRESSION

ry

2 cy

z e

Crrryy

prererss

| SIGNATURE

DATE

3 cn

7 e

y

Form 771-24 (11-7)


CHAPTER 2

ENFORCEMENT OF PROGRAM REGULATIONS

Every I/M program is different, and as a result the regulations which
must be enforced will be different as well. There are basic questions that
can be asked though, about almost any program (see Table 2-1). The initial
focus of covert auditing activities then should be to identify information
which can answer these questions.

FOCUS OF ENFORCEMENT ACTIVITIES

In order to structure an enforcement-oriented covert auditing pro-
gram, management must be clear on not only the overall program goals
which must be addressed (Table 2-1), but also how those goals could be
compromised. Figure 2-1 presents a way of analyzing this process.
Working from the goals of the program, it is possible to identify the
discrepancies which could compromise those goals; working from these
discrepancies, it is possible to identify program practices, or indica-
tors, which can cause these discrepancies. It is these 1andicators, or
program practices, which should be the focus of an enforcement-oriented
covert auditing program. Generally, these program practices will not be
detected through formal periodic auditing.

Omission of Test

Probably one of the more serious problems in a decentralized program
is the 1ssuance of a compliance certificate or sticker without a test.
This program abuse will clearly lead to false passes; it also has the po-
tential for undermining the credibility of the program. Formal periodic
auditing cannot document this practice; 1t can only uncover suspicious
data totals (e.g., an extraordinarily low failure rate). Even covert au-
diting may not be able to uncover this abuse when it 1s performed private-
ly for customers that station personnel know well; however, well-planned
visits with a solid cover (see Chapter 5), can still uncover a lot. Be-
cause a station may omit a test on any vehicle, theoretically either pas-
Sing or failing vehicles may be introduced into a station. However, if
passing vehicles are introduced into a station, auditors must be with
the vehicle so they can provide first hand information that certain parts
of the test were not performed. If failing vehicles are introduced, and
their condition before and after a test can be verified, auditors do not
have to observe the test. In many State and local programs, passing a


TABLE OF CONTENTS

List of Tables

List of Figures

Executive Summary

Disclaimer
Acknowledgment
CHAPTER 1 THE ROLE OF UNMARKED VEHICLES IN DECENTRALIZED
I/M PROGRAMS
Introduction
Comprehensive QA in a Decentralized I/M Program
Covert Auditing - The Role of Unmarked Vehicles
Outline of a Covert Auditing Program
Structure of this Manual
CHAPTER 2 ENFORCEMENT OF PROGRAM REGULATIONS
Focus of Enforcement Activities
Prioritizing Stations:
An Ald towards Documentation and Deterrence
The Issue of Entrapment
Penalties, Publicity and the Durability of
Deterrence
CHAPTER 3 COLLECTION OF DATA ON PROGRAM OPERATIONS
Coordination with Enforcement-Oriented Auditing
Specific Types of Studies
CHAPTER 4 EXAMPLES OF RESULTS FROM CURRENT AUDITING PROGRAMS
Prevalence of Detrimental Practices
Deterrence Through Publicity
Development and Disposition of Cases
CHAPTER 5 PROGRAM ISSUES
Program Elements
Two Model Programs
REFERENCES
APPENDIX A SHOP GUIDANCE
APPENDIX B BACKGROUND MATERIAL FROM ONGOING I/M PROGRAMS

ai

aii

iv

vi

vii


TABLE 4.1

SUMMARY OF COVERT INVESTIGATIONS IN THE COLORADO I/M PROGRAM

(January 1 - November 22, 1985)*
Improper Improper Other
Issuance Visual
of a Sticker Inspection

Visual Failures
(total attempts = 535) 7.5% 4.9% 6.3%
Emission Failures
(total attempts = 381) 3.7% 0.8%** 5.9%
Total
(total attempts = 891) 6.8% 3.2% 7.2%

NOTE - Percentages do not total, as multiple violations may occur from
one audit. These totals also do not reflect violations which
were addressed 1n an office conference only.

* Includes only those investigations which have gone to formal
hearings by 11/22/85.

** Excludes one vehicle which had both emission and visual failures;

n = 356,

4-2


OT-z

Pigure 2-3
California Station/Mechanic Summary
CEFSOTRENT CF CONSUMED AFFAIRS
GLREAL GF AUTCPOTIVE REPAIR

STATICA / MECKANIC EVALUATICN REPCRT PAGE 14
CATA CCLLECTEC 2nd GLARTER 1985
SIAIICH.bavE EIELE_CEEICE SE
COC OOOO D
ONC VOC OOD
FHOAE: (418) CODCOD
- oe eT TESTSTQNISIICS eee S-IESTST@1ISIICS 22 LL. -
TOTAL FRCAT (© X CEV FROM CISTPICT AVERAGE ) TCTAL FRONT ( X DEV FROM CISTRICT AVERACE
TeS1S | EAIL EAIL_EATE tf. £0. TESIS £AlL EAIL-BATE = EGZBOLE §=cO-EDUL
gta a 141 er 7 SELOW 726 - 2 5 69 10 AECVE 723 “17
FEC b 144 «} 7? SELCw tue -2c 5 60 2C ABCVE 72? 717
® CERTS BY 1 DEV FROM IDLE SFEED AUMBER OF FERCEAT NC. ANC XA
INACCURATE STATEWICE DEVIATICN RETESTS TESTS W/TK TESTS bITH
PREV EMIS CILLTION TIC A TEST 404 CO TC 113 RFP OF NC PATCHING
USlB£S01s QVEEESE...  BLECS.3.CEY = IUBESHCLE TeRESHCLD. TTEST.WL.
STA # 2 C.2 74 35.0 oO 23.5 1 19.4
rec # 2 C2 74 35.6 c 23.8 1 19.9
PERCENTAGE ERRCKECLS NC. ane x XA TEST X PULTIFLE NC. ANC X CF
UNCERHCCD ENTRIES CN WITH FAST WITH NO TESTS FAILED TESTS
Tuc CURACY FUNCT a f8_ BEEALES.. Beeales._ Se WIIBCEETS._
STA # 1é 79 oO 2 74.9 0.0 4.9 c.0 ? 9.3
vEC & Ve 80 c 274.5 0. 4.5 6.0 3 9.2
tus 27 5 4

NOTE- See Appendix B-4 for descriptions of headings in this table.


GUIDANCE MANUAL

Using Unmarked Vehicles in
Decentralized I/M Programs

Submitted to

Emission Control Technology Division
U.S. Environmental Protection Agency
2565 Plymouth Road
Ann Arbor, MI 48105

March 1987
46924.00/51,51A

Submitted by

Engineering-Science
Two Flint Hill
10521 Rosehaven Street
Fairfax, Virginia 22030


TABLE 5.2--Continued

Program Element

Alternative 1

Alternative 2

Vehicle Set-up

Audit Documenta-
tion/Verifica-
tion

Shop Facilites

Enforcement
System

Publicity

Data Collection

and Program
Assessment

Minimize vehicle set up,
except to make sure that
failing vehicles really
fail.

Document only that informa-
tion necessary to support
the enforcement actions
contemplated.

Use any enclosed space with
basic tools and an emission
analyzer.

Utilize office conferences
as much as possible, rather
than extensive hearings and
Prosecutions.

Use any means possible (es-
pecially training and for-
mal audits) to remind sta-
tions that covert auditors
are in the field. Do not
divulge under any circum-
stances the size of the
operation.

At the beginning of a pro-
gram, start the development
of baseline data. Send out
at least some failing vehi-
cles in audits which will
not lead to enforcement
actions to determine if
false passes are a problem.

Set up at least one 1981 and
later vehicle to fail; other
vehicles should be set up to
test for particular program
practices.

Document all data relevant
both to potential enforcement
actions and to the program
practices investigated.

Develop a shop which supports
the set-ups chosen; equip the
shop so that 1981 and later
vehicles may be adjusted or
modified.

Depending on available legal
authority, experiment with
different approaches to de-
termine what methods provide
the most deterrence.

Along with the suggestions
of Alternative 1, identify
and publicize areas in
which vehicle owners can

play a part.

Develop baseline data and
then track the progress of
the program.



What may be needed 1s a baseline, 1.e., a quantitative description
of ongoing successes and flaws at a point in time. In other words, gather-
ing data 1n many cases provides an opportunity to determine how a program
1S improving as a result of a covert auditing program. The issue of base-
line development 1s discussed in more detail 1n Chapter 5.

Managers of many I/M programs feel fortunate to have enough resour-
ces for the basic activities of program operation; extra data collection
and analysis may be considered to be a beneficial but expendable part of
a program. While at times special studies can be resource-intensive, in
many cases they can be integrated into existing operations (see Chapter
5). Regardless, probing into the operations of an I/M program can give
insights 1nto not only what 1s happening, but why. Without reliable, ac-
curate information on what 1s really happening in I/M programs, managers
cannot really manage. If, for example, real failure rates are low and
emission reduction effectiveness is less than the effectiveness planned
in the program design, finding out why is the first step toward correct~
ing the problem.

SPECIFIC TYPES OF STUDIES

Gathering information about the operations of an I/M program can ei-
ther be through a special study or on a continuous basis. In any event,
studies must be organized. The following discussion presents a brief des-
cription of how to approach the four subjects presented in Table 3-1.
Table 3-2 presents some potential comparisons that can be generated as a
result of focused data collection analysis.

Failure Rates

Investigations of failure rates have been discussed in some detail in
Chapters 1 and 2. If false fails and false passes are being investigated,
Passing vehicles and failing vehicles are usually needed. Special docu-
mentation/verification may be needed if the adjustment (or non-adjustment)
of certain engine parameters 1s to be studied. Failure rates are of
interest for all of the subgroups listed in Table 3-2 (different types of
analyzers, service outputs, etc.) particularly if a program wants to
identify where (if anyplace) low failure rates are concentrated.

Repair Cost

As with I/M effectiveness studies, repair cost studies need data on
the types of maladjustments and poor maintenance found in an area. Model
year groupings and types of service outlets are key groups to investigate.
For 1981 and later vehicles, vehicle preparation is critical.

Warranties
Warranty studies will of course be focused on dealerships, and will

involve only late model vehicles. Again, information on the sorts of fail-
ures that dealerships actually face is useful.


effective covert audit. (One state even reported effective audits using
a State vehicle with a State seal on the vehicle's door.) However main-
taining cover must not be taken lightly. Even if some audits can proceed,
two problems can surface 1f a station detects an audit and "plays along":

o the deterrent effect of being caught 1n a covert audit
will only be felt by those who fail to pick up the clues.

o information gathered by auditors will not be representative
of program operation, and neither the auditors nor the
program management will know it.

Accordingly, good planning is essential to maintaining cover, and
program managers must therefore consider how an auditor's cover could be
lost. Among the clues to stations which should be considered are:

the auditors themselves

auditors' dress

the vehicles themselves

the combinations of auditors and vehicles
the auditor's story (is it plausible?)

oo0000

To the extent the vehicles and auditors are known, an auditor's cover is
suspect. A combination of people, vehicles, etc., is desirable. In addi~
tion, auditors must develop a routine for themselves and managers of audi-
tors must review every part of an investigation package (the combination
of auditor/dress/vehicle/story) to make sure that suspicions are not arou-
sed. Periodic review of results in the field 1s helpful to adjust the
overall program to eliminate potential problems.

A checklist of questions is provided here as Table 5-1 which can be
used to assess the reasonableness of the audit approach.

Passing or Failing Cars

This issue has been discussed at some length in previous chapters,
and what makes sense may vary from program to program. Ideally, a program
should initiate a data gathering study to see what practices are occurring
in the field. A selection of vehicles aid in characterizing program
operations (see Chapter 3). In the absence of any data about the practices
occurring in a specific problem, using failing cars at least in part is
a good starting point. Given the false pass phenomenon and the ability in
Many cases to avold the need for direct observation of the inspection,
failing cars may provide an information baseline to build on. In addition,
even if there are doubts in a particular jurisdiction about the legality
of using failing cars in a specific case, using cars in a data collection
effort can never be considered entrapment.

Personnel

Position descriptions for auditors, and 1n some cases, supervisory
auditors are presented in Appendix B-3. It should be noted that the Cali-
fornia position description provided 1s for a Bureau representative who
works with licensed stations and auditors.

5-2


TABLE 2-3 (Continued)

Notes to Table 2-3

High passing rates may indicate a combination of phenomema occurring
simultaneously; those phenomena which may contribute are noted with
ao",

Direct observation 1s needed to demonstrate a specific practice. A
vehicle may still be a failing vehicle and the station could be cited
for passing a failing car. If at is desired to determine whether a
specific station 1s, for example, adjusting engine parameters in a pre-
repair phase, then a program must develop a way to document the status
of those parameters before and after the test. See Chapter 5 under
"Shop Issues."

Badly maintained probes may be checked through formal periodic audit-
ing as well.

Data probably available only from certain automated analyzers.

Generally, only with manual analyzers. Direct observation is not
necessary if passing or failing status of the car can be demonstrated
before and after the test.

2-13


fail cars that should be passing. False fails would therefore have to
be investigated using vehicles set to pass. To a certain extent, compu-
terized analyzers (generally those which meet BAR84 or similar specifica-
tions) make such fraud more difficult to perpetrate. The pass/fail
decision on the vehicle 1s made by the analyzer, and in many analyzer
systems, integrated automatic data collection systems allow the tracking
of 1ssuance of compliance certificates.

It appears though, that as of 1986, a more significant problem in de-
centralized I/M systems is stations that incorrectly pass vehicles which
should fail ("false passes"). It may not be stated categorically that this
situation exists 1n all decentralized I/M programs, but rather this prac-
tice 1s well enough established that investigating the phenomenon should
be considered in the establishment or modification of a covert auditing
program. Some programs have noted that when an inspection fee is high
enough, a significant portion of stations and/or inspectors will purposely
pass cars in order to capture fees 1n as short a time as possible. To
test the false pass phenomenon discussed above, clearly vehicles must be
set to fail. Vehicles must also be set to fail to investigate the pre-
repair phenomenon.

Observation of Inspections: Documentation of Results

Actual observation of inspections may or may not be necessary to be
able to document the results of covert investigations. How auditors are to
conduct themselves and what data they are to collect will depend in large
Part on program regulations. It also may depend in part on how much prac-
tically can be seen during an investigation. Frequently customers are not
allowed in service bays. In such cases, auditors may be forced to wait
outside, and first hand observation of certain parts of a test (such as an
EGR functional test) could be impossible.

Other Issues

Observation of inspections is directly related to other structural
elements of an auditing program. For example, direct observations of
inspections may not be necessary 1f the program has a well-equipped shop
which can thoroughly document a vehicle before and after an inspection.
Similarly, if the question of interest is whether a failing vehicle will
be passed, the documentation of the vehicle can probably be accomplished by
agency shop facility personnel. An auditor needs little training to
simply take a vehicle to a licensed station and bring back to the shop
results of the inspection. An approach of this sort may be an aid to an
implementing agency 1n terms of personnel; fewer requirements for qualifi-
cations makes hiring auditors easier. Details of these and other issues
are discussed in Chapter 5.

Table 1-2 presents two alternative approaches to developing a covert
auditing program and how the issues discussed above may be addressed in
different ways. There are of course advantages to each alternative, and
other alternative approaches could easily be generated. However, any co-
vert auditing program must address the issues presented here, as well as
many other considerations which are presented in detail in the following
chapters.

1-5


NEW YORK STATE
DEPARTMFNT OF MOTOR VEHICLES
DIVISION OF VEHICLE SAFETY SERVICES

REGIONAL OFFICE PROCEDURE FOR CONCEALED IDENTITY INSPECTION OF
INSPECTION STATIONS

Material Received;

1. Written or orai instruction from Sr. Automotive Facilities
Inspector to perform a concealed identity inspection of a specified
inspection station,

OR

2. For downstate regional offices, printout of inspection stations
from the Technical Services Bureau. Printout will indicate
information such as:

a. excessive inspections conducted daily and monthly:
h. excessive passes;

c. excessive failures;

d. excessive adjustments or repairs on certain items, etc.

Operations:

1. Preparation for inspection.

a. Obtain and sign for cash from CI fund to cover the cost of the
inspection and any minor repairs that may be necessary,

h. Obtain a vehicle, registration plates, integrated windshield
stickers, and inspection sticker from Regional Office to use
during the inspection,

c. Check vehicle to be used.

1, Registration must correspond with plates.

2. Inspection sticker should indicate expiration or
near~expiration,

3. Re sure that the vehicle is in good repair.


REFERENCES

Cabaniss, John M., Jr., "Overview of I/M Activities and Issues,"
Proceedings - Mobile Source Issues in the 1980s, SP-52,
Midwest Section, Air Pollution Control Association,
Pittsburgh, PA 1984.

Carhart, Bruce S., “Real World Approaches to Inspection/Maintenance
Quality Assurance: Field Auditing Issues," Paper No. 85-40.5,
presented to the 78th Annual Meeting of the Air Pollution
Control Association, Detroit, Michigan, 1985.

Pienta, Walter J., et al, "Quality Assurance in Decentralized I/M
Networks: Analyzer Measurement Error and Field Performance,"
Paper No. 852110, Society of Automotive Engineers, Warrendale,
PA, 1985.

Schroeer, John, "Covert Vehicle Modification" (revised), Colorado
Department of Health, Denver, CO, November 1985.

U.S. General Accounting Office, "Vehicle Emissions Inspection and
Maintenance Program is Behind Schedule," GAO/RCED-85-22,
Washington, DC, 1985.

U.S. Environmental Protection Agency, "A Review of Control Strategies
for In-Use Vehicles," EPA-460/3-74-021, Office of Mobile Source
Air Pollution Control, Ann Arbor, Michigan, 1974.

U.S. Environmental Protection Agency, “Information Document on Auto-
mobile Emissions Inspection and Maintenance Programs," EPA-400/2-
78-001, Office of Air and Waste Management, Washington, DC, 1978a.

U.S. Environmental Protection Agency, "Motor Vehicle Emissions Inspection/
Maintenance Information Kit," EPA-460/3-78-013, Office of Air, Noise,
and Radiation, Washington, DC, 1978b.

U.S. Environmental Protection Agency, "Guidance on Quality Control for
Inspection/Maintenance Programs," EPA 460/3-82-006, Office of Mobile
Source Air Pollution Control, Ann Arbor, MI, 1982.

Wallauch, John R., “The California Approach to Vehicle Inspection/Main-
tenance," Paper No. 831213, Society of Automotive Engineers, Warren-
dale, PA, 1983.


TABLE A-1

SUGGESTED TOOLS FOR A BASIC SHOP

Basic Tools

Standard: open end; box end; hand wrenches 3/8" - 1"

Metric: open end; box end; hand wrenches 9 mm - 23 mm
Standard: closed sockets 1/4" - 3/8" - 1/2" drive
Metric: closed sockets 1/4" ~ 3/8" - 1/2" drive

socket drivers 1/4" - 3/8" - 1/2" drive

Standard tip screw drivers
Philips tip screw drivers
Hammers
Files
Hack saw
Air operated socket drivers

3/8" hand held

1/2" impact driver

Special Tools
For Pre-1981 Vehicles:

Timing light with advance control

Tach and Dwell Meter

Vacuum~Pressure Tester

Vaccum Pump

Volt Meter

OHM Meter

Distributor Wrenches

Exhaust Gas Analyzer

Engine Analyzer (scope) (all of the above can be
incorporated in enginer analyzer)

For 1981 and later vehicles:
All of the above plus -
Computer Scanner

Digital OHM Meter
Needle type OHM Meter


FIGURE 1.1

STRUCTURE OF THE MANUAL

CHAPTER 1

INTRODUCTION

Concepts
Goals
Methods

|

!

CHAPTER 2
ENFORCEMENT-ORIENTED
COVERT AUDITING

Issues
Structure
Approaches

{

\

CHAPTER 3
COVERT AUDITING
FOR_DATA-GATHERING

Types of Studies
Relationship to Enforcement

I

CHAPTERS 4 & 5
PROGRAMS RESULTS & ISSUES

Reported Data

Practical Considerations
Setting Up A Program

Model Programs

APPENDICES