A11101 111715

NBSIR 79-1764

Image Pattern Recognition in

Industrial Inspection

Gordon J. Vanderbrug

Roger N. Nagel

Automation Technology

Mechanical Processes Division

Center for Mechanical Engineering

and Process Technology

National Engineering Laboratory

National Bureau of Standards

Washington, DC 20234

Issued September, 1979

•J) Q

NATIONAL BUREAU OF STANDARDS

NBSIR 79-1764

IMAGE PATTERN RECOGNITION IN

INDUSTRIAL INSPECTION

fcaStonat Bureau of'StaWSFtw FEB 8 1880

no

&CU-00

u s u

c .

Gordon J. Vanderbrug

Roger N. Nagel

Automation Technology

Mechanical Processes Division

Center for Mechanical Engineering

and Process Technology

National Engineering Laboratory

National Bureau of Standards

Washington, DC 20234

Issued September, 1979

U.S. DEPARTMENT OF COMMERCE, Juanita M. Kreps, Secretary

Luther H. Hodges, Jr., Under Secretary

Jordan J. Baruch, Assistant Secretary for Science and Technology

NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director

TABLE OF CONTENTS

1. Introducti

2. Visual Ins

3. Visual Ins

4. Other Appl

5. Conclusion

on

pection Systems

pection for Robots

ications

s and Future Trends

-i -
1 . INTRODUCTION

Image analysis and pattern recognition can play a large role in

the industrial arena. In this report we present a state of the

art survey illustrating the application of these technologies to

the problem of visual inspection. The important aspects of these

technologies for visual inspection are:

Digitization and Quantization - the image must be converted

to a form where it can be processed digitally.

Compression - techniques such as contour coding which reduce

the amount of data.

Enhancement - techniques which manipulate gray levels or

modify histograms so as to transform the image into a form

which is more suitable for processing.

Segmentation and Feature Extraction - techniques such as

thresholding^ edge detection* and template matching which

are used to segment the image and produce features.

Classif ication-techniques such as discriminate functions*

decision trees* and inspection procedures which are employed

to make decisions.

References 1 and 2 are good introductions to image analysis and

pattern recognition. Reference 3 contains a discussion of the

visual inspection problem* while reference 4 discusses the use of

sensors in industrial automation app 1 icat ions.

The use of visual inspection in industry is relatively new* thus

the applications we have surveyed include production systems (installed on the factory floor)* prototype systems (designed for

use in production)* and feasibility studies (performed as a research effort). In feasibility studies the needs for computational simplicity and cost have been de~emphasi zed and further

development would be needed to produce a production system.

The motivating factors in applying visual inspection systems

vary. In some cases the task is simple* but the speed required

(several decisions per second) precludes the use of human inspectors. In other cases the job is boring and human inspectors do

not perform well (and don't want the job). In addition to the

human factors mentioned above* there are a host of economic and

environmental motivations for the use of automated visual inspection. A further and increasingly important need for visual inspection* arises in the need to provide sensory feedback for

robot manipulators. In the case of a robot* a human is by definition not available* yet often there is a need to perform visual

tasks. The lack of a vision system on commercially available

robots has both limited the range of robot applications* and increased the cost of many robot installations.

-2-
Section Two of this report surveys a variety of stand alone visual inspection systems developed by industry. The third section

surveys the work on robot "eyes" developed for inspection problems In the fourth section/ a list of other applications for

automated inspection is presented. Finally/ in the last section

we present our conclusions/ and an assessment of the future role

for visual inspection in industry.

-3-
Visual Inspection Systems

The explicit inspection of parts and assemblies in the manufacturing process has been estimated to represent 10% of the total

labor cost. This 10% figure is second only to the cost of assembly, which is estimated at 22% of the labor cost C43. Thus there

is a high potential payoff in cost when an automated visual inspection system can be installed.

In this section we present a brief overview of several visual inspection systems. In general these systems match a set of predefined features found by image analysis and pattern recognition

techniques with a stored range of values for these features. The

end result is to label the part as acceptable and allow it to

pass. or to reject the part as defective. The vision system is

usually connected to some part handling equipment* such as a transfer line* which removes rejected parts from production.

For each of the app 1 icat ions# the

describing the manual process* or

between the acceptable and the defec

the hardware configuration used in

techniques used in the inspection p

will provide information on performance

rob lem wi 1 1 be stated

by ind i cati ng the differ

ive parts. We will indi

the sys tern* and describe

oce dure . If possible*

by

the

we

l

-4-
2 . 1 General Motors Chip Alignment

Figure 1 is an assembly consisting of a large heat

cup on the corner of the heat sink* and a Dari

which is bonded to the surface of the heat sink C53

bly is a component in the High Energy Ignition sys

Motors automobiles. The manual inspection process

semblies is illustrated in Figure 2 and consists of

steps. The assemblies are automatical ly fed int

table where an operator 1) visually (through a micr

mines the position and orientation of the chip# 2)

any gross defects in the chip/ 3) manually manipu

electrical test probes over the base area of the ch

sticks# and 4) initiates the lowering of the probes

contacts and the automatic electrical test process.

Laboratories developed the vision programs and GM

Development produced a production line system to

process.

sink# a weld

ington IC chip

The assemtem in General

of these asthe following

o a rotating

oscope) deterinspects for

lates a set of

ip using joy

onto the chip

GM Research

Manufacturing

automate this

The processing is illustrated in Figure 3. It begins by histogramming the orientations of an edge detector. The maximum value

is taken as the approximate orientation of the chip on the heat

sink. Chips that have too much tilt for the automatic test

equipment are identified. At the same time that the edge detectors are applied# corner detectors are used to locate possible

corners of the chip. A square template# equal to the size of the

chip# is used to search the space of possible corners# to determine the actual position and orientation of the chip. At this

point# chips that are off the edge of the heat sink or too close

to the weld cup are identified.

Simple contrast detection schemes are used to verify that the

chip is not broken# undersized# or fractured (see Figure 4). Thus

the system determines the position and orientation of the chip#

rejects chips that are structurally defective# and directs the

alignment of the test probes over the chip contacts. A schematic

of the system is shown in Figure 5.

The prototype system used a 50 X 50 pixel camera# with the field

of view twice the width of the chip. Sixteen gray levels were

used. The system was equipped with an infrared illumination

source. A production line system shown in Figure 6# was installed in a Delco Electronics plant in January of 1977.

-5-
Load

Station

Test Probes ““ Microscope Viewer

X, Y.6-

Joy Sticks For Manual

Probe Alignment

Figure 2

Darlington Electrical Test Probe Station

Fig. 1 Locating an Integrated Circuit Chip

Camera 1 mage

of Integrated Digitized Orientation Determined through All Possible Actual Corners Appearin Windows of a Relational Circuit Chip Picture Entered Gradient Direction Corners of the Part Mask Applied at the on Heat Sink into Computer Analysis are located Known Orientation 0 OProbability of Orientation

• 45

°

o';;;- o

0

°" > 45

°

ntation

Figure 3

6
7
2 . 2 Western Electric Series 700 Connector

Figure 7 shows the Western Electric Series 700 Connector and the

internal contact C63. They are used for making quick# reliable#

water resistant splices of two or three insulated conductors.

The connector consists of a slotted "U" shaped contact# preassembled into a two piece sealant-f i 1 led plastic housing. A splice

is made by inserting the wire into the housing and compressing it

to make the contact cut through the wire insulation# and thus to

make the connection.

The craftsman normally does not make a visual or electr

of a splice. Hundreds or thousands of individual

splices make up a single cable splice# which is typical

after completion. Currently# 30% of the parts are

manually# a percentage based on the cost of inspection

duct quality. One hundred million of these parts ar

tured each year. Figure 8 shows various categories of

parts# such as a missing or misoriented contact# and in

sealant.

ical test

connector

ly buried

inspected

and proe manufacdef active

sufficient

The inspection process is based on identifying

characterize good connectors. The connector

viewing it from the side using transmitted light,

appears as a dark upside down M U"» and the

sealant can be determined by the amount of transmi

the appropriate areas.

features which

is inspected by

The contact

presence of the

tted light in

Nine features are used to characterize a connection. Average intensities in the areas "B"# "C M

» and "F" (see Figure 9) are used

for the sealant. The area N is used for normal i zat ion. The contact position is determined by the five measurements shown in the

lower half of Figure 9. These measurements are made using a median filtering operation on three adjacent lines of columns.

Inspection is carried out by an algorithm involving these nine

features and thresholds that have been determined to separate the

good and the bad parts.

A prototype system (Figure 10) was developed at Bell Laboratories

using a solid state# 100 X 100 resolution TV camera. Since the

amount of data to measure the sealant parameters is minimal# this

resolution was determined by the accuracy requirements for the

contact position.

Presently# Western Electric

parts handling components,

p 1 emented on an LSI-11, and

eras. The performance goal

is integrating the inspection and

The inspection component is being imwill operate two GE - TN 2200 camis one part per sec. for each camera.

-8-
I

Figure 7 Western Electric

Series 700 Connector
2. 3 General Electric Syntactic Analysis System

General Electric has developed a system for inspecting industrial

parts which uses syntactic pattern recognition C73. The system

uses a finite state grammar to analyze the median curves of parts

such as screw threads or springs. The system» shown in Figure

11* is notable for its degree of generality and the amount of

processing that it does in hardware. It consists of a rotating

glass disk which back lights the object as it passes in front of

a RETICON solid state linear imager. The rotation of the disk is

synchronized with the scan clock of the imager. The output is

threshalded to produce a binary signal corresponding to the part

silhouette* such as that shown in Figure 12.

The outline of

hardware by lo

dirt on the gla

filtering* and

sive lines. Th

the part silhouette is obtained in special purpose

eating edge transitions. The presence of dust and

ss and gaps within the silhouette are handled by

by requiring adjacent edge transitions on succese edge silhouette determines the median curve.

Straight line approximations are done sequentially by an algorithm which looks for the next data points inside an envelope.

The envelope is allowed to change slightly* based on the length

of the line and the average slope. The algorithm has been implemented with a standard 181 arithmetic logic unit chip* and is capable of line fitting to data paints at 2 X 20*000 points/sec.

The next step is to perform the grammatical analysis using syntactic pattern recognition. The terminal symbols of the grammar

are numerals chosen to represent straight line segments of various lengths and orientations* as shown in Figure 13. For example* the symbol 9 represents a short segment oriented in a northeast direction. The grammar determines a finite state

machine which is used to parse the median axis* and thus* to accept or reject the part.

An example sequence of

represent one down s part of Figure 14. The

is in the right part

small noise burst which

the network.

non-terminal symbols

lope and one up slop

state transition ne

of the figure. The

is indicated in the

(i.e. numerals) that

e is shown in the left

twork for this example

down slope contains a illustration and in

The grammar is

finite state*

extensive - it contains 1360 productions,

but has been implemented with a stack for

It is

the last

10-
three symbols; so that limited backtracking can be performed.

The prototype system for the syntactic analysis has been implemented on an Interdata 7/16 minicomputer; and can process 15

parts/sec. Currently the hardware preprocessor is capable of

processing 5 parts/sec.

The finite state grammar approach is sufficiently flexible to be

a candidate approach for inspecting any manufactured part where

the median curve is a good representation. One difficulty in the

syntactic approach is in determining the grammar. There has been

some work in grammatical inference; and this work may some day be

useful for syntactic pattern recognition applications to industrial inspection.

Figure 11

- 11 -
12
2.4 Lockheed Shell Inspection

Lockheed Palo Alto Research Laboratory performed a study which

demonstrated the feasibility of digital image analysis techniques

for the automatic inspection of 105mm projectiles using x-rays

C81. The defects which appear on the x-rays are cracks# cavities# porosity# and base separation. The steps taken in the

feasibility study are as follows.

The original digital image of the projectile is shown in the

upper left of Figure 15. The image has a 500 micron resolutiion

and nine bit intensity quant i zati on. The upper right image of

Figure 15 shows markings of the shell boundary used to adjust the

shell orientation. The lower left image of Figure 15 outlines

the region which will be searched for defects. In the final image of this figure# this region has been transformed by a “flat

field" correction algorithm. This algorithm fits a second order

polynomial to the raster-scanned intensity values# subtracts the

fit values from the actual intensity values# and scales the

result. Some hints of possible defects begin to be visible in

this image.

Individual defective picture points are shown in the left of Figure 16 as bright points. They are determined on a point by point

basis. This determination is made by comparing certain statistics on the intensities inside a small neighborhood to the

statistics on the intensities inside a larger neighborhood.

In the final image (Figure 16 RIGHT) the individual defective

picture points have been associated into distinct defects. This

process is a simple region growing algorithm with a threshold (in

this case 25 picture points) for the minimum allowable defect

size. Features such as size# length# width# and orientation are

computed and used to classify the defects. The details of the

classif ication have not been reported.

The 500 micron resolution was sufficient to locate the above defects# but was not sufficient to check for base separation of the

projectile. For this reason# the base region was rescanned at a higher resolution for that part of the inspection process.

The feasibility study was considered a success and a prototype

system is under development.

- 13-
Figure 15 (Above)

Figure 16

14
2.5 Recognition Systems* Inc. - Optical Power Spectrum Analysis

Recognition Systems* Inc. has adopted an approach toward automatic inspection based on optical power spectrum analysis C9D. An

optical computer takes the Fourier Transform of a part. The important features of the resulting power spectrum are sensed by a solid state photodetector* and fed into a digital computer for

analysis. A block diagram is shown in Figure 17.

One of the applications that Recognition Systems has demonstrated

is the inspection of hypodermic needles (Figure 18). A hypodermic needle diffracts a laser beam through a lens to produce the

power spectrum. The needle is placed in the front focal plane of

the lens* and the detector is placed in the back focal plane*

where the power spectrum is formed. The detector consists of

rings and wedges to measure energy in various regions of the

power spectrum. A solid state electronics package consisting of

an amplifier* a multiplexor* and an A/D convertor delivers the

signal to a computer.

This approach is based on certain properties of the Fourier

Transform that make it possible to predict the structure of the

transf orma t i ons of various objects. In particular* circles are

transformed into circles* and lines are transformed into lines

that pass through the origin at 90 degrees to the object line.

In a smooth picture the falloff in moving out from the origin is

very gradual* while in a busy picture it is much more rapid.

Directional biases in the picture show up as directional differences in the rate of the falloff. This is why the datector consists of a semicircle of annular rings and a semicircle of

wedges.

Figure 19 shows both a good and a defective needle point* and

their respective power spectra. The defective needle contains a horizontal edge that is not present in the case of the good needle. This edge results in a vertical component of diffracted energy in the detector. By examining the pattern of the wedge intensities* the presence of the horizontal edge* and hence the defect* is detected.

The computer executes the inspection algorithm and controls the

material handling subsystem. The overall throughput of the system is 10 needles per second.

It is reported that the savings over manual inspection cost is

- 15-
sufficient to pay for the machine in less than one year. The

needle point system was installed in a factory approx imately two

years ago* and is the first industrial inspection application by

Recognition Systems. Presently they are working on applications

to photomask inspection* paper printability determination* and

fabric defect inspection. The -fabric inspection (Figure 20) will

produce a quality grading of the product.

The power spectrum of a high quality fabric exhibits a regularity

that is not present in the power spectrum of the low quality fabric. The individual lobes of the pattern for the low quality

fabric are more deformed. Note the effect of a double-thread in

the woven cloth on the non-control lobes in the power spectrum of

the weave pattern in Figure 20.

Figure 21 is a prototype drawing for the

plans are for 100% inspection of webs o wide at the rate of 100 yards per minute,

decisions across the width of the web*

defects will be assigned* and an overall

cloth passing through the system will be

inspection system. The

f fabric up to 48 inches

Based on individual

point counts for various

grade for each piece of

determined.

- 16-
GENERIC FACTORY SYSTEM

DIGITAL

COMPUTER

PRODUCT w optical

COMPUTER

Figure 17 (Above)

Figure 18

l-.V'

materials

HANDLING

Optical computer takes the

‘ouch’ out of hypodermics

sts

U&we* teens mMjrr

t&oet >€xr nesi/ tanm c=

VPOXVBZTBe

(ptesruEt neaxmcues.) c=

Multipiexw

(ferilMHtS

Aupunces

at HGffenoHtnwc IHSTAmANeOUS pu* AMPimjcx a yyI V . .11, I'.. ml

\ :l \

I »ll 1 1 I. I I’ lll. Ill ..I ( ,1 Vr.ll,

r.iui

i • i 1 1 in. i M ..I i ti l, . nt, V II. I'.. ml

Figure 19

17
WOVEN CLOTH GRADING

Figure 20

Fabric Inspection

TYPICAL WEAVE PATTERN

Figure 21

18
2.6 Hitachi Shadow Mask Inspection

A color picture tube has a shadow mask at the back of its front

panel. It is made of a thin iron plate on the surface of which

is a grid of several hundred thousand small rectangular holes.

The holes are formed bg contact-exposure printing and chemical

etching. The patterns of the master plate are frequently damaged

in production or in the printing process* hence the master plate

must be inspected. Manual inspection of a master plate is shown

in Figure 22. An automatic visual inspection system for shadow

mask master plates has been developed by Hitachi* Ltd. C103.

A master pattern is a 610 x 800 mm plate with rectangular patterns approx imately .10 x . 60 mm in size and approximately . 10 to

.20 mm apart. Figure 23 shows the kinds of defects and the dimensional tolerances in the inspection process. Defects include

chips* projections* and pin holes in the patterns* as well as dimensional errors.

The image acquisition component of the experimental equipment

(Figure 24) consisted of a solid state linear image sensor* a light source* a sliding table with driving motors and control

circuits* and a microscope. The table is moved to form a raster

scan of the master plate. The image sensor is a RETICON linear

array* which provides a resolution of 7 micrometers under the

conditions that the clock frequency is 5 MHz and the magnification of the object is 10 times.

The video signal is threshalded and processed in a minicomputer.

The processing is done by comparing successive patterns* and by

examining features of individual patterns. Features used include

area comparison* maximum width comparison* distance between patterns* and width change within a single pattern. Each field is

128 x 512 picture points* and contain 3 pattern stripes.

A production line machine (Figure 25) for the automatic inspection of master plates has been constructed. It is reported that

the machine operates at 200 stripes/sec. and the performance is

equal to that of skilled inspectors.

- 19-
Figure 22

No. Name Feature Limit

size

1 chip-1 fio 10pm

2 chip-2 tn rf 10pm

3 projection 10 pm

4 pinhole O 3

5 missed [; c 10 pm

6 too large

it

'D 10pm

7 too small 10pm

8 dot

°=t 10pm

9 bridge

width

ir ZJLZ 10pm

10 position

error -3E 10pm

Figure 23 Types of Defects

Figure 24 (Left)

Figure 25

20
2. 7 Hitachi Relay Switch Inspection

Hitachi* Ltd. has built a contact points of relay

as displacement of the c splashes hinder the perf

spection criteria have be

fects. For example* the

along the X and the V axi

The system configuration

vertical reflected ligh

nification* and a movable

place when the table is s

machine

switches

antac t

ormance

en estab

d isp lac

for th e automat ic inspection of

C1U. Defects (Fi gure 26) s uch

point* sera tche s* and weld ing

of the relay swi tch Visual in1 ished for each 0 f these d eement s hould be les s than 0. 1mm

s.

(Figure 27) consists of a TV camera* a

t source* and an objective lens for magtable. The inspection process takes

tationary.

The image processing for a single contact point is done in a sequence of windows of the entire image. Within each window a threshold is chosen* so that processing is done on a binary image. Thresholds are determined on the basis of the maximum and

minimum gray levels within the window.

Figure 28 shows a window (labeled "inside frame") which

corresponds to the smallest acceptable contact size. Previous

processing has determined the contact center* which is used to

fix the location of the window. The search for scratches and the

adherence of an alien substance is restricted to this region.

The search itself consists of looking in 7 X 7 neighborhoods for

sets of above threshold picture points which form a chain across

the neighborhood. In a similar manner* examining the contact

point for a welding splash is done by searching with a different

size neighborhood along the perimeter of the larger window in

Figure 28.

Figure 29 shows the appearance of the production line machine.

It employs two cameras* and is capable of inspecting 32 contact

points in 5. 4 sec.

-21 -
spring sheet

(a) Displacement (b) Deformation

(c) Scratch (d) Adherence of alien substance

(e) Welding splash (f) Collapse Figure 26 (Above)

outside frame

welding splash

(intermittent movement)

Figure 27 (Above)

Figure 29 (Above)

Figure 28 (Left)

22
3. Visual Inspection for Robots

A visual inspection capability is a potentially important component of a programmable manipulator or robot. Typically# a human factory worker performs several implicit inspection tasks in

a materials handling job. It is reasonable to expect the worker

to inspect for: the presence or absence of a part# completeness

of a part# presence of major defects such as cracks# burrs# and a variety of simple defects. If a robot is to perform this implicit inspection aspect of materials handling it needs sensory

feedback. Vision sensors using the techniques of image analysis

and pattern recognition are now appearing on prototype robots and

in feasibility studies.

Even when the robot is not performing an inspection task# sensory

feedback is important. Sometimes the identity of the part is not

known# such as in a foundry where a number of different castings

could be on a conveyor belt.

Often, however, the identity of the part in a material handling

situation is known. but its position and orientation are not

known. A vision system provides the information necessary for a robot to acquire the parts.

There is still another category where sensory feedback for robots

is important, where the approximate position and orientation of a part is known. Here the choice is to achieve higher accuracy in

part location through the conveyor# assembly line and special

purpose machinery, or to equip the robot with sensory feedback.

The latter is potentially more economical.

-23-
3. 1 General Motors Part Recognition

General Motors Research Laboratories has developed a vision system that can determine the position and orientation of complex

curved objects in gray-level noisy scenes C12*133.

The system is able to recognize parts such as connecting rods and

steering knuckles that are lying flat on a conveyor belt which is

at a known distance from the camera. No special illumination is

used. The system stores a model of each part that it is able to

recognize. Additional models for parts are added to the system

by showing the part under favorable conditions.

Figure 30 is a 256 X 256* 16 gray level image of a connecting

rod. The Hueckel edge operator produces the 400 edge points

shown in Figure 31. Edge points are linked together using a local proximity and orientation criteria to form chains (Figure

32). The program puts an asterisk at the end of open chains and

shades the interior regions of closed chains. Curve fitting

methods are used to replace the chains by what are called concurves* which are sequences of straight lines and arcs. Figure

33 shows that nine chains have been replaced by four concurves*

which consist of 19 straight lines and arcs. A program uses this

data to obtain model concurves (Figure 34)* which are stored in

the data base so that future connecting rods can be detected.

Figure 35 shows the digitized picture of six parts on a conveyor

belt* the edge points* the concurves* and the models superimposed

on the gray level picture. The problem of matching the concurves

of a part to the right model amounts to finding the translation

and rotation which maps the model into the concurves of the part.

There are two steps in the matching process. First* one model

concurve is matched against one image concurve to determine a tentative transf ormat ion. Second* the transf ormat ion is checked

in a global manner by matching a representation of the total

model data with the total image data. The search aspect of the

matching process is aided by a heuristic which measures the

matching likelihood for each model/image concurve pair that is

possible. This heuristic is based on general character ist ics of

the concurves. The system can work when the parts are occluding*

such as shown in Figure 36. The effort was a feasibility study.

Presently work is going on to reduce the computat ional and

storage requirements.

-24-
Figure 31 (Below)

Digitized picture of

connecting rod

Figure 30 (Above)

Result of applying Hueckel edge operator on Fig. So Output: 400 edge points containing

location, direction, and average intensity on both sides.

Result of linking edge points to form chains.

Output: 9 chains. The program puts a at the end of open chains and shades the interior region of closed chains.

Figure 32 (Left)

Result of * curve fitting to form concurves.

Output: 4 concurves which consist of 19 straight lines and arcs.

Figure 34

Model concurves obtained by program from table

of input geometric data.

25
Figure 35

Six parts on a conveyor belt. (a) Digitized Picture,

(b) Edge Points. (c) Concurves. (d) Models superimposed

on gray-level picture.

26
Figure 36

Six parts on conveyor belt with considerable occlusion,

(a) Digitized Picture. (b) Edge Points. (c) Concurves.

(d) Models superimposed on gray- level picture.

27
3. 2 SRI Internat ional

SRI International has been working in the area of vision systems

for inspection and for materials handling C14/15D. Their system

begins operation by thresholding the image of a part* such as one

of the castings shown in Figure 37. A connectivity analyzer

determines the outline of the part* and a set of features is computed. The features used include: perimeter of the outline*

square root of the area* total hole area* minimum radius* maximum

radius* average radius* and a ratio of perimeter to area. These

features were chosen because they are invariant with respect to

translation and rotation.

The four castings s hown in

shown in Figure 38. All of

the stab 1® states. The sys

c i s ion tree* such as the

then use d to recogn ize indi

of the tree is taken t

between the classes In th

and hap pens to b e th e separat

s the connecting ro

cal i per from the rest of

see that the minimum rad i us

Rec ursively repeat ing th i dec ision tree.

Figure 37 have seven stable states* as

the features are computed for each of

tern automaticaly computes a binary de—

one shown in Figure 39. This tree is

vidual parts. The feature at the top

o be the one which has the largest gap

is example* that feature is called x3*

minimum radius in the outline. It

d« the piston sleeve* and the brake

the stable positions.

feature performs

s process generates

One can readily

this separation,

the entire binary

SRI has also used their system for inspection app 1 ications such

as for the inspection of the lamp bases shown in Figure 40. The

system provides the user with the capability to determine the

edges of the bases and the contacts* and provides programming

tools to assist him in implementing an inspection procedure. The

procedure must determine that: 1) there are two contacts* 2)

that the spacing between them is correct* and 3) that the' position of the two contacts with respect to the base is correct. In

Figure 41* only one part passes inspection.

SRI has mounted a solid state camera in the hand of their Unimate

robot (Figure 42). The camera is mounted parallel to the axis of

the hand. The mode of operation to precisely locate an object is

to: 1) position the camera over the object* 2) use the previously

described image processing software to find the object* 3) move

over a fixed distance equal to the separation of the hand and the

camera* and 4) move straight down to the object.

-28-
Figure 37 (Left)

Four Foundry Castings

Figure 38 (Right)

Seven Stable States

Connecting

Rod

Cylinder

Head (1)'

(On Side)

Piston

Sleeve ( 1

)

(Upright)

Brake

Caliper ( 1

)

(Lying Flat)

Cylinder

Head (2)

^"(Upright)

Piston

Sleeve (2)

(On Side)

Brake

^"Caliper (2)

(Lying on Side)

Figure 39 (Left)

Decision Tree

SLEEVE 2 CALIPER 2 29
Figure 42

Camera on Unimate

30
3. 3 Auto-Place Robot

Figure 43 shows an industrial robot made by Auto-Place Inc. At

the Robots II Conference in 1977 they presented a demonstration

that showed how their robot could modify its behavior from sensory input. The sensory input comes from a solid state camera

mounted horizontally on top of the robot. Light is deflected

from a horizontal work table to the camera. In the demonstration* the robot picked up a pair of dice that had been randomly

placed on the work table. It also determined the throw of the

dice* by simply adding up the black points (obtained by a simple

threshold algorithm) which correspond to the dots on the face of

the dice. In an earlier demonstration Auto-Place used two robots

working together with a camera to sort a deck of cards into suits

(see Figure 44). The first robot would pick up a card from a

card tray with a suction grip and hold it up to a camera mounted

separately from the robot. The camera and computer system determine the suit by looking at one of the suit symbols* thresholding

it* and using a simple shape criterion. If the card did not fall

into the proper categories (club* diamond* heart or spade) the

robot was instructed to turn the card over. If again the suit

could not be determined* the robot dropped the card on the floor.

When a suit was determined* the robot which picked it up* placed

it into the sorted pile of hearts or diamonds as appropriate.

Uhen a club or spade was found* the card was passed to the second

robot and appropr iately placed in the proper sorted pile. AutoPlace has not yet reported an industrial application of their

robot with a camera. They are* however* currently working on an

inspection application of vision for the Jeep Corporation. This

inspection system will consist of ten cameras linked to a microcomputer. The system will analyze the body-mounting pad heights

on an automative frame. This analysis determines the amount of

shimming required at each body- mounting location* in order to

insure a flat mounting plane for the body.

-31 -
Figure 43

Figure 44

32
3. 4 National Bureau of Standards

The Automation Technology Program at the National Bureau of Standards is developing a vision system for its robot C173. The vision system consists of a solid state camera, a strobograph ic

light source, and an 8-bit microprocessor. The camera is mounted

obliquely at the wrist of the robot, so that its field of view

covers a region extending from inside the finger tips out to a

distance of one meter. The light source flashes a plane of light

parallel to the wrist of the robot into the region. Figure 45

shows the configuration of the camera, flash unit, and the field

of view. The resulting image sensed by the camera has a variable

scale (shown in Figure 46) due to the angulation of the camera.

The net result is that objects far away from the wrist appear in

the coarsely calibrated region of the image, whereas objects

close to the wrist appear in the finely calibrated region. The

plane of light produces an image with bright line segments. The

computer system computes a run length encoding representat ion of

the line segments. The interpretation algorithm uses only the

line segment data, and based on the fact that tr iangulation gives

range data, the slope of the lines indicate the orientation of

the object, and the end points of the lines provide information

on the edges to be grasped. Figure 47 shows the camera and flash

unit mounted on the robot, and Figure 48 illustrates the line

segments obtained from some simple objects. The computer can

control the duration of the flash, and therefor® the contrast in

the image. The camera system provides an image with 128 by 128

pixels, and is quantized to eight bits.

The software component of the system is still being developed. A

typical application is in materials handling, such as a three

step sequence for the task of acquiring an object which has been

randomly placed on a table. First the table is scanned in a plane which is approx imately parallel to the table. The object

will appear in the image, and range values for its locations can

be computed. In general, these values will only be estimates of

the objects' location, since the computation will be done in the

course resolution part of the scale. Next, the estimate of the

location can be used to move closer and take a second view of

the object. This more precise determination of the location can

be used to signal the control to move the arm above the object in

preparation for grasping it. Finally, a third look can be used

to verify conditions prior to grasping. Figure 48 shows a set of

simulated images which illustrate this sequence.

The intent of the system is to demonstrate the concepts for a highly flexible low cost, vision system. By using the plane of

light and a dedicated microprocessor, a hard wired data reduction

-33-
scheme

permits

the rob

ontained in the vision sensor has been achieved,

the analysis of vision data to take place in real

t moves about in its environment.

This

time as

-34-
V

DISTANCE

IN

CENTIMETERS

CAMERA

Figure 45

PIXEL COLUMN

35
Figure 47

36
Figure 48

A series of pictures which illustrates the three step part

acquisition sequence for a box.

37
4.0 Other Applications

In section 2 and 3 we presented a sampling of application in

visual inspection systems. There are other applications at other

institutions. For example* work on printed circuit board inspection has been done at the University of Missouri and at Princeton

University. Work on computer vision for industrial applications

has gone on at Stanford University* at Hitachi Central Research

Laborator ies* at Osaka University* and at the University of Rhode

Island. This is not meant to be an exhaustive enumeration.

In this section we present a list of other applications of visual

inspection to industrial problems. Information on applications

by private companies is often difficult to obtain. Often the

very fact that a company is considering

Nonetheless* we do have a ] present. There are no del

However* the diversity of the list

Kenneth White of Proctor and

list. Additional < 18 and other sources.

pr opr i etary

cations to

avai lab le.

provoking,

the compilation of this

tained from, reference

ng an appl i cation i s t of ind ustr i al app li —

Is 0 f th e app 1 icaton s ist is i tself thou gh t

Gamb le c ol lab orated i n ppli cati ons were ob —

-38-
1 grasshopper contaminat ion of string beans

2. frozen food quantity count variations

3. dust detection on LSI substrates

4. bottle fatigue detection* to prevent explosions

5. thermometer inspection

6. label inspection on plastic bottles

7. golf ball label inspection

8. IC chip wire bonding

9. measurement of the effectiveness of a window defogger over

time

10. automatic debarking* initial timber cutting* sorting of

1 umb er

11. measurement of biscuit height so that packaging equipment

doesn / t jam

12. unthreaded nut detection

13. button inspection for the correct number of holes

14. sorting of porcelain seals by shape and conf iguration

15. automatic closed loop control of extruded gelatin sausage

casings

16. hot roll steel web width control

17. particle inspection of pharmaceutical liquids

18. closure integrity inspection for pharmaceuticals

is the rubber seal inside the cap provided by the suppi i er?

is the cap tightened?

is it not cross threaded?

is the seal over the cap properly in place?

19. blister pack inspection - is there one pill or blister?

-39-
20. china plate inspection for pinholes and hubbies on the plate

21. automated separation of whole almond nut meat from debris«

shells* and damaged meat.

-40-
5. 0 Conclusions and Future Trends

What are some common character ist ics

lems that we have discussed in detai

is that the majority of the applicati

tion> in the sense that they wer

specific problems.

of the inspection prob1? A first observation

ons were hard automae specific solutions to

One exception to this is the syntactic pattern recognition

approach to inspecting parts* such as springs and screws*

where the median axis is a good representation of the part.

The primary task in applying this technique to a new application is the development of a new grammar. The optical

power spectrum approach is also somewhat of an exception to

the hard automation observation. Here the user must develop

a classif ication procedure based on the 32 wedge and the 32

ring features of the photodetection.

The parts recognition work at General Motors and at SRI

Internat ional are also exceptions in that new part systems

can be added to these systems. These systems are primarily

intended for material handling applications as opposed to

visual inspection. The barrier of hard automation for visual inspection has been addressed at SRI* in that they have

developed interactive programming aids to assist the user in

developing new inspection procedures.

The systems that are in (or are nearly in) the factory are

for applications with high production volumes. This tends

to require high inspection rates: one per second for the

Series 700 Connector* 5-15/sec. for the screws and springs.

High inspection rates result in a big emphasis on data

reduction. Image processing applications inherently begin

with large amounts of data. One guideline to keeping a handle on the amount of data is to use the coarsest acceptable

resolution. This can be determined in part by the smallest

dimension that must be measured. In part it seems that some

experimentation is required to determine the coarsest acceptable resolution.

Similar things are true for

plications discussed used bi

spection application used a

quant i zation. Many of

nary images. Only the

fine quant i zat ion.

the

shell

ap~

in-

-41
Probably the best parameter in estimating what inspection

rate may be achievable is the complexity of the inspection

process itself. How difficult will it be to identify the

features that are necessary to distinguish between the good

and the bad parts? How complex will the segmentation process be? What will the c omputat i onal complexity of the decision procedure be?

One heuristic in estimating the complexity of the inspection

process is the amount of difficulty that a technically competent person* who has no previous knowledge of the component being inspected* experiences in understanding the

manual procedure. Ideally* the manual procedure should be

rather obvious.

A good inspection applica

techniques for controll

the part should be viewed

known orientation. Using

ditions results in proces

times* such as with th

the second dimension can

the part. Sometimes s tion, moving the part is

Control of the illuminati

niques* can reduce the c true for control of the o and reflectivity) of the

ten these parameters are

inspection designer.

tion will make effective use of

ing the environment. If possible*

from a fixed location* with a an array de tec tor under the se consin g only a single image. Somee s pring and the sc rew appli cat ion*

be effective ly prov ided by moving

uch as with the sha dow mask inspecthe only opt ion.

on* such as with lighting techomp lexity of the ta sk. The same is

pt i cal parameters (such as color

part being inspect ed* although ofnot in the d omain o f control of the

What is the future for automated visual inspection systems?

The future is good! The future is good because there exists

a substantial* fundamental understanding of image pattern

recognition* and basic research in this area continues. The

future is good because the semiconductor manufac tur er s have

provided a variety of linear and array sensors* and functioning cameras using these devices. These solid state cameras are potentially rugged and reliable* and only cost in

the neighborhood of one to three thousand dollars. The future is good because the microprocessor is substantial ly

reducing the cost of computat ional power. Also the cost of

memory is coming down.

So the -future is good. Nonetheless* the future is not assured. The production engineer and the -factory manager must

be brought to the point were they view automated inspection

-42-
systems as one more tool to accomplish their jobs. This

means that adequate attention must be paid to the details of

the production engineer's concerns for performance and cost.

A track record of successful applications must be built up.

That track record is only beginning to appear.

-43-
'A (REV. 9-78) 1EPT. r

~ 1. PUBLICATION OR REPORT NO. 2. BIBLIOGRAPHIC DATA

79- 176A

(NBS) SHEET

4. TITLE AND SUBTITLE

Image Pattern Recognition in

7. AUTHOR(S)

Industrial Inspection

8. Performing Organ. Report No. Gordon J. VanderBrug and Roger N. Nagel

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, DC 20234

12. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street, City, State, ZIP)

11. Contract/Grant No.

13. Type of Report & Period Covered

Same as 9

15. SUPPLEMENTARY NOTES

[_J Document describes a computer program; S F- 185, FIPS Software Summary, is attached.

16. ABSTRACT (A 200-word or less factual summary of most si ghi ficant information. If document includes a significant bibliography or

literature survey, mention it here.) Most manufacturing processes require a visual inspection of some aspect of the process.

In the most straight forward applications, a vision system is used to inspect for part

completeness, or to check for flaws in the manufacturing product. In more sophisticated use of vision systems the inspection task may be secondary to tasks such as part location, identification, or determining part orientation. In general these tasks are important when the vision system is used in conjunction with a robot manipulator. In

robot systems, vision is needed to allow the robot to acquire, manipulate, and inspect

parts without the need for elaborate fixturing, or complex part delivery systems. The

labor cost of manual inspection, and the high cost of special purpose part delivery

systems for robots have led many manufacturers to investigate vision systems for use

in manufacturing. Digital image processing and pattern recognition are providing the

basis for a growing number of attempts to achieve an automated vision system.

This paper begins with a brief historical perspective on image processing and pattern

recognition. Next a series of state of the art examples of visual inspection systems,

and then robot vision systems is presented. The paper contains a list of other areas

in manufacturing for the application of vision systems, and concludes with an assessment. of the future of vision systems in manufacturing.

17. KEY WORDS ( six to twelve entries; alphabetical order; capitalize only the first letter of the first key word unless a proper name;

separated by semicolons)

Automation, inspection, image processing, manufacturing, pattern recognition,

robotics, vision systems

18. AVAILABILITY [~X| Unlimited 19. SECURITY CLASS 21. NO. OF

(THIS REPORT) PRINTED PAGES

1 I For Official Distribution. Do Not Release to NTIS UNCLASSIFIED

( | Order From Sup. of Doc., U.S. Government Printing Office, Washington, DC

20402, SD Stock No. SN003-003-

20. SECURITY CLASS

(THIS PAGE)

22. Price

[ | Order From National Technical Information Service (NTIS), Springfield, V A. 22161 UNCLASSIFIED $4.50

U SCOMM- DC