Feasibility of using intelligent video

for machinery applications

B Y TODD M . RUFF

R

ESEARCHERS AT THE

National Institute

for Occupational

Safety and

Health (NIOSH) studied methods

to prevent workers from being

entangled in machinery used at mining

operations. An analysis of mining accidents in

the United States, which resulted in a fatality or

permanent disability from 2000 through 2005, showed

438 severe incidents that involved contact with machinery

or equipment, an average of 73 per year. The researchers

determined that the most common machinery involved in

these severe accidents was conveyors. In addition, a significant portion of the accidents occurred during machine

maintenance and repair. Researchers focused on improved

methods to prevent unintentional machine start-up during

maintenance activity and methods to detect workers near

moving machine components. One new technology that

showed promise for this application was intelligent video.

Popular in the surveillance and security industries, these systems usecamerasand computerized video analysis techniques

© IMAGESTATE & STOCKBYTE

to automatically detect the presence of people in preset zones

within the camera’s field of view. A preliminary study was

initiated to determineif this technology could reliably detect

the presence of workers in hazardous locations near machinery. The possible advantages of using this technology

include improved detection zone demarcation and

improved ability to distinguish between hazardous and

nonhazardous proximity, compared with conventional

proximity sensor techniques. Initial tests to detect a person near a conveyor system in daylight conditions showed

promising results. A description of the technology, test

procedures and results, implementation challenges, and

future research needs are discussed in this article.

U.S. Government work not protected by U.S. Copyright.
0

Active and Intelligent Sensing Technology

Although the total number of mine worker fatalities in

the United States has been on a downward trend [1], the

proportion of these accidents that involve mine machinery

and mobile equipment has consistently been significant.

Researchers at NIOSH and other organizations have been

concerned with the interaction of workers with machinery

and the number of severe accidents classified as struck by

or caught in [2]–[5]. These accidents include workers

entangled in rotating machinery, struck by moving

machine components, or run over by mobile equipment.

An analysis of accident data available from the Mine Safety

and Health Administration (MSHA) [6] was conducted

for the purpose of better understanding the problem and

the scope of machine-related accidents.

On average, severe accidents that involved a worker and

machinery or haulage equipment comprised 42% of all

severe accidents at mining operations.

Figure 1 shows the top 20 machine

types involved in these accidents,

with conveyors listed most often

(13%).

0 10 20 30 40 50 60 7

Number of Accidents 

Conveyor (All Types) 

Rock or Roof Bolting Machine 

Ore Haulage Trucks 

Front-End Loader 

Milling Machine 

Continuous Miner 

Crane, Derrick 

Load-Haul-Dump 

Crusher, Breaker, Mills 

Bulldozer 

Forklift 

Trucks (Not Ore Haulage) 

Dragline, Shovel, Excavator 

Shuttle Car 

Powered Hand Tools 

Screen 

Drills 

Longwall Parts (Jacks or Chocks) 

Shop Power Tools 

Pump 

1

Mining machinery involved in severe accidents, 2000–2005, n = 438

(top 20 machines).

The activity of the worker

during the accident was also of interest (Figure 2), with a significant portion of injuries and fatalities occurred

during maintenance and repair of

machines (24%).

Machine Maintenance/Repair 

Handling Supplies or Material 

Haulage Truck 

Surface Equipment NEC 

Continuous Miner 

Roof Bolter, NEC 

Utility Truck 

Hand Tool (Not Powered) 

Bulldozer 

Front-End Loader 

Conveyor Belt (Not Riding) 

Walking/Running 

Hand Tools (Powered) 

Inspect Equipment 

Roof Bolter, Drilling 

Unknown 

Moving Equipment 

Longwall, Shear, Plow 

Move Power Cable 

Roof Bolter, Inserting Bolt 

0 20 40 60 80 100 120 

Number of Accidents 

2

Worker activity at the time of the accident, 2000–2005, n 1⁄4 438 (top 20 activities).

Also of interest was

that, in approximately 40% of the

fatalities involving stationary equipment such as conveyors and crushers,

lockout/tagout procedures either were

not followed or were not adequate for

the particular machineinvolved [7].

Although many factors contribute

to machinery accidents, one possible

approach for preventing caught-in

and struck-by accidents entails better

methods of detecting hazardous situations. Guarding, emergency stop

devices, and safe control system design

have all contributed to a decrease in accidents, but improvementsarestill needed. Activeand intelligent sensing technology that can better detect human presence in hazardous areas

could decreasetheseaccidents further.

NIOSH researchers have identified intelligent video systems (IVSs) as a potential tool that could reduce accidents

involving machines. The IVSs are used by the security and

surveillance industries [8] and in some safety applications

such as detecting a distressed person in a public pool [9].

These systems use standard video cameras connected to a

computer to automatically analyze the video signal and

provide notification if a person enters a predefined zone

within the camera’s field of view. Other functionality

includes the detection of items left behind, the removal of

items, and a worker down. The IVSs employ familiar and

accepted camera technology. These may eliminate the need

for additional sensing systems and may reduce false alarms

when compared with other sensorbased technology [2], [3].

Intelligent Video Systems

An IVS uses a computer to perform a

sophisticated analysis of video camera

images. The main motivation for its

development was to provide automatic recognition of certain events

(intrusion, loitering, and theft) for

surveillance and security applications, thus reducing the work load

associated with the constant observation of video monitors by security

personnel [10]. The analytical functions available on most commercial

IVSs include detection of intrusion,

loitering, and abandoned or removed

objects, as well as vehicle/person

tracking and vehicle counting. The

detection of personnel entering or

leaving a specified zone is a common
function of these systems, and it is the application that

prompted NIOSH researchers to study the use of this

technology for machine safety.

The IVSs analyze video frame by frame to detect changes

in thescene or image[11]. For instance,a baseline or nonevent

video image for a particular camera is taught to the analysis

system (this is a dynamic process, and recalibration is constantly occurring atintervals specified by software parameters).

Ifa person then walks into thescene, thesystem detects thisas

a change in the baseline image. Boundaries can be defined

within the image to limit detection to a certain perimeter.

When detection occurs within this perimeter, alarms can be

activated and/or control signals generated. Parameters can be

set to alarm only on objects of a certain size and shape. Filters

can be employed to ignore events of short duration or the

effects of shadows. Thermal imaging and low-light cameras

can be used if needed for nighttime operation.

The following list summarizes the potential advantages

of theIVSs for machinery applications. Thetests have been

designed by NIOSH researchers to verify these features.

The IVSs may offer

n a relatively low cost, low maintenance, stand-alone

detection system that uses available camera technology

n a vision-based object recognition with increased

ability to discriminate between hazardous and nonhazardous situations

n a well-defined hazardous zone demarcation

n the ability to monitor large areas with a single camera

n increased difficulty in circumventing, disabling,

or ignoring

n many alarm output options, including a simple

audible alarm in an operator’s station to prompt a

check of the video, remote on-site alarms or beacons, generation of machine control signals, and/or

personnel paging and e-mail

n the capability to digitally record video footage of

alarm events.

Preliminary Tests

A preliminary evaluation of intelligent video technology

was conducted by NIOSH researchers during a 2007 pilot

project. A small rock conveyor was used as a test platform

(Figure 3).

3

The test conveyor at NIOSH Spokane Research Laboratory.

An eight-channel IVS manufactured by Arteco

Vision Systemsand three different types of outdoor cameras

were purchased for evaluation. Several different scenarios

were tested: perimeter protection to detect the presence of a

worker near the conveyor; monitoring small areas on the

conveyor to detect the presence of fingers, hands, and arms

in hazardous areas; monitoring the removal of machine

guards or enclosure lids; and detecting a worker lying

motionless on the ground. The tests described here were

primarily conducted using an inexpensive outdoor color

camera(420 lines of resolution, 4.3 mm lens).

At the time of this report, only tests conducted with a

stationary belt during daylight conditions in fair weather

were completed. The tests in dark or stormy conditions and

tests with moving material on the belt were still pending.

The tests during daylight were conducted at different times

of the day and with different degrees ofcloud cover to determine the effects of sun position and cloud shadows. The

tests to detect the presence of a worker near the perimeter of

the conveyor were conducted to determine the reliability of

the system as a proximity warning device. Another intrusion test was conducted to determine the ability of the system to detectappendages or limbs in hazardousareas.

The detection zones were drawn by simple point-andclick actions to define the corners of the detection area in a

particular camera’s video window. These zones are indicated by green boundaries (Figure 4).

4

The detection zones for whole-body detection tests.

If a person entered a

zone, the boundary turned red, indicating an alarm event.

At the same time, an audible alarm was activated, and a

control signal was generated by using a relay interface.

The tests were conducted in the morning (9:00 a.m.) and

again just after noon (1:00 p.m.) to observe the effect of

different sun positions and resulting shadows. Each test

wasconducted with thesame test subject, first in light-colored clothing and then in dark-colored clothing to determine the effects of differing clothing contrasts against the

asphalt background. The test subject entered the zone

several times and at several different locations for each test.

The IVSs also remained activated between the morning

and afternoon tests to seeif falsealarms were generated.

Whole-Body Detection

With high-sensitivity settings, the system reliably

detected a person entering either theleft or rightzone (Figure 4), regardless of clothing color. No missed detections
occurred during approximately 80 test events at various

points along the detection zones. This detection reliability

was seen for both the morning and afternoon tests. Consistent detection occurred when at least one half of the body of

the test subject was in the zone. There were no false alarms

during the tests, but there were ten false alarms in the 4-h

period between the tests because of birds and insects flying

directly in front of the camera lens. The sensitivity settings

were lowered, and filter adjustments were made to decrease

false-alarm rates,and thetest was repeated on another day.

With the new settings, detection of a person was not as

reliable. Clothing color seemed to be a factor: dark clothing

was detected 55% of the time and white clothing 90%. No

falsealarms wereseen between or during thetests. A tradeoff

exists between reliable detection of a person and false-alarm

rates. Fine tuning of the filters could have resulted in better

results. However, these tests showed that the system, as configured, could probably not be used to directly control the

machine, e.g., automatic shutdown when someone gets too

close. The settings that allowed high reliability also

produced false alarms that would not be tolerated if they

resulted in frequent interruptions in production. However,

the proximity warning functions may be useful in providing

an alert to control room operators or other personnel to indicate the presence of workers in hazardous areas. The alert

could prompt a plant operator to check the video monitor,

and the operator could then make a decision to ignore the

event, shut down the machinery, or warn the worker via

loudspeaker or othercommunication.

Appendage Detection

The next test was to determine the reliability of the video

system in detecting smaller portions of the body encroaching on critical areas of the conveyor, e.g., the belt and pulley areas. Figure 5 shows the detection zone drawn over the

tail pulley area.

5

The tail pulley zone for appendage detection tests.

This detection analyticcould only beactive

when the conveyor was not running—the moving material

and belt would constitute constant changes within the

detection perimeter and would result in false alarms. Thus,

the detection and alarming would be activated during

maintenance, when theconveyor was motionless. Ifa worker’s hand, arm, or body was detected in this scenario, an

alarm control signal would be provided to the mainconveyor controls to prevent start-up. In this way, the system could not be restarted if a worker was performing

maintenance or if maintenance was completed but not all

personnel were clear of the area. The false alarms in this

scenario could be tolerated. The tests showed that the false

alarms were short in duration and the cause was evident in

the video, allowing quick correction and resulting in only

short delays in the execution of machine start-up.

The tests were conducted in the morning and afternoon

at several different locations along the detection area. The

sensitivity settings were set high to detect small objects in

the detection zone. The tests were run to detect fingers, a

hand (to the wrist), an arm (past the elbow), and head/

shoulders leaning over the belt (Figure 5). For each body

part, ten detection tests were conducted in the morning

and ten in the afternoon. The fingers were detected with

80% reliability in the morning test and 60% in the afternoon. A hand was detected with approximately 60% reliability in the morning and afternoon (sporadic detection

occurred at some locations but was considered unreliable

detection for these calculations). An arm over the belt, or

head/shoulders leaning over the belt, was detected 100%

of the time in both tests. Figure 6 shows the blob view

that the computer is analyzing during the arm detection

test.

6

The corresponding blob view after processing.

In this view, black indicates areas that have not

changed from the baseline, whereas white areas indicate

new features or changes in the scene.

Sometimes the test subject’s shadow was detected before

the subject’s body entered the zone. This occurred about

10% of thetime, most often during the morning tests when

shadows werelonger. This was not seen asasubstantial deficiency—the presence of moving shadows was usually associated with a true-alarm condition. If a person’s shadow

generated a false alarm that prevented the start-up, the

video monitor could be checked to verify the source of the

alarm. Twelve false alarms occurred during the 4-h period

between the two tests, and the alarms were most often

caused by birds or insects flying directly in front of the

camera. These false alarms were of short duration and could

be minimized by adjusting the short event filter on the

video analysis system. Again, further adjustment of the

software settings could improve detection of fingers and

hands but most likely at the cost of increased false alarms.

The detection of larger portions of a person’s arm was more
reliable. That and whole-body detection may be all that is

needed for reliable prevention of machine start-up.

Missing Machine Guards and Worker Down

Finally, limited tests were conducted to determine whether

the IVSs could detect the removal of a machine guard or an

electrical enclosure cover (removed object analytic) and an

injured worker on the ground (abandoned object or worker

down analytic). The reliability of detecting a missing

machine guard or an open electrical enclosure was low in

most circumstances. The background contrast issues and

false alarms when a person stood in front of the machine

guard or enclosure contributed to the low reliability. Furthermore, a separate camera would need to be dedicated for

monitoring the guarded area to provide enough detail, i.e.,

the same camera could not be used for the overall perimeter

protection and for missing machine guards. The reliability

did increase when high-contrast placards were attached to

the machine guard to provide more positive detection.

However, problems were still seen when a worker stood in

front of the machine guard, effectively making the system

think that the guard was not visibleand thus missing.

Detection of a motionless person lying on the ground was

possible, but false alarms were common when a person stood

still in the same detection area. Different camera angles and

adjustment of software parameters resulted in better performance, but false alarms still occurred because of changing light

conditions or similar-sized objects left in the detection area.

It is possible to define the shape and size of the object to be

detected so that a person’s body lying on the ground can be

distinguished from someone standing upright. Such refinement requires much experimentation, and reliability could

still be questionablein congested work areas.

Conclusions

NIOSH researchers are studying methods to reduce struckby and caught-in accidents, involving machinery at mining

operations. This is a challenging problem—in many circumstances, workers must be near moving machinery to perform

their jobs. Innovative safety interventions are needed to further

decrease accidents without adversely affecting productivity.

Monitoring hazardous areas near machines by using the IVSs

may be one possible solution, but limitations must be understood and anticipated during implementation. Although more

tests are needed, some general conclusions can be drawn about

the possible applications of intelligent video for stationary

machine safety

n One promising application may be the use of IVSs

tosupplement lockout/tagout procedures during

maintenance, i.e., monitoring disabled machinery

and preventing start-up if someone is detected in

a hazardous area.

n The current systems could be used to detect the presence of workers in hazardous locations near operating

machinery, but implementing automatic machine

controls would have to be done with caution and

only for functions that could tolerate occasional false

alarms. Providing alarms that increase situational

awareness during production for plant operators may

be a more beneficial and realistic application.

n The detection of missing machine guards or other

safety equipment will require more sophistication

and intelligence than what was possible with the

IVSs tested here.

n	 Worker-down detection may be possible with the

IVSs if false alarms are tolerated. This analytic is

typically applied to a large field of view, resulting

in low object resolution and making human body

recognition difficult.

n	 The reliability of many IVS applications may be

increased using multiple cameras to provide different views of the same scene or, as in the case of stereovision, to provide three-dimensional information.

n	 More tests are needed to determine the effects of rain,

snow, fog, dust, and dark conditions on the reliability

of theIVSs forall theaforementioned applications. The

outdoor operation of the system has been reliable in

security and surveillance, but reliability requirements

increase for safety applications, and environmental

effects must be well understood. Additional tests will

also be needed to conduct a thorough comparison with

alternativesensor-based technologies.

Acknowledgments

The author thanks Richard Rains, John McCallum, and

Edward McHugh of NIOSH and Justin Willett, formerly of

Arteco Vision Systems, for their assistance with installing the

test conveyor and cameras and testing the IVS. The findings

and conclusions in thisarticle have not been formally disseminated by NIOSH and should not be construed to represent

any agency determination or policy. Mention of a specific

company or product does not imply endorsement by NIOSH.

References

[1] V. Kecojevic, D. Komljenovic, W. Groves, and M. Radomsky, “An

analysis of equipment-related fatal accidents in U.S. mining operations: 1995–2005,” Saf. Sci., vol. 45, pp. 864–874, 2007.

[2] T. Ruff, “Recommendations for evaluating and implementing proximity warning systems on surface mining equipment,” NIOSH Rep. of

Investigations 9672, DHHS (NIOSH) Publication 2007-146, 2007.

[3] M. Venem, J. Shutske, and W.Gilbert, “Testing and creation of a

safety system to disengage the PTO of a tractor,” Appl. Eng. Agric.,

vol. 22, no. 1, pp. 5–12, 2006.

[4] T. Backstrom and M. Doos, “Problems with machine safeguards in automated installations,” Int.J. Ind. Ergon., vol. 25, no. 25, pp. 573–585, 2000.

[5] W. Schiffbauer, “An active proximity warning system for surface and

underground mining applications,” in Proc. Longwall USA Int. Conf.,

Pittsburgh, PA, June 7–9, 2005, pp. 1–8.

[6] Mine Safety and Health Administration (MSHA). (2008, Apr. 9). U.S.

Dept. of Labor, Part 50 Accident, injury and illness reports [Online].

Available: http://www.msha.gov/STATS/PART50/p50y2k/p50y2k.HTM

[7] Mine Safety and Health Administration (MSHA). (2008, Apr.9).

Fatal alert bulletins, fatalgrams, and fatal investigation reports

[Online]. Available: http://www.msha.gov/fatals/fab.htm

[8] J. Edwards, “The unblinking eye,” Electron. Des., pp. 38–40, Apr. 27,

2007.

[9] S.Sookman, “Thrown into the deep end,” Security Technol. Des.,

vol. 17, no. 6, pp. 58–59, 2007.

[10] D. Marman. (2008, Apr.17). How video analytics is changing the

world of security [Online]. Available: http://www.videoiq.net/products/

resources

[11] A. Umamakeswari and A. Rajaraman, “Object based video analysis, interpretation and tracking,” J. Comput. Sci., vol. 3, no. 10, pp. 818–822, 2007.

Todd M. Ruff (TRuff@cdc.gov) is with the National Institute of Occupational Safety and Health in Spokane, Washington.