Reference

NBS

Publications

AlllOM MTflb37

NBSIR 78-1576

Final Report - Proximity-Vision

System for Protoflight

Manipulator Arm

James S. Albus, Ph.D.

Project Manager

Automation Technology

Mechanical Process Division

Center for Mechanical Engineering

and Process Technology

National Engineering Laboratory

National Bureau of Standards

Washington, DC 20234

Issued August, 1978

Prepared for Marshall Space Flight Center

Huntsville, Alabama 35812

John L. Burch

EC 25/MSFC Representative

100

.056

78-1576

^^onal Barwa of

1 1

NBSIR 78-1576

I r i FINAL REPORT - PROXIMITY-VISION

SYSTEM FOR PROTOFLIGHT

MANIPULATOR ARM

James S. Albus, Ph.D.

Project Manager

Automation Technology

Mechanical Process Division

Center for Mechanical Engineering

and Process Technology

National Engineering Laboratory

National Bureau of Standards

Washington, DC 20234

Issued August, 1978

Prepared for Marshall Space Flight Center

Huntsville, Alabama 35812

Under Order Number H-30093B • August 1 2, 1 977

John L. Burch

EC 25/MSFC Representative

tA-'

• —

U.S. DEPARTMENT OF COMMERCE, Juanita M. Kraps, Secretary

Dr. Sidney Harman, Under Secretary

Jordan J. Baruch. Assistant Sacretary for Science and Technoiogy

NATIONAL BUREAU OF STANDARDS, Emaat Ambler, Director
. d
Contents

Introduction

lechiiical Approach

Solid State TV Camera

Infra—red sensors

Display-Control Console

Computer Interface

Conclusions and Recommendations

Picture Quality

Slip Ring F<equ ir emen ts

Computer Analysis

Recommended Future Development
VT ' '-la

<1 'i &.$ ' f ^ -' r

'- •

V . If ^ <t X-^ yB

''.. rr,,0 I .^T-i r’n'.'*- . i

•

5 3 e • 1 C

‘ -J ij

• ijitia li*® Uftsvr,r,:0:3 ! « r> ft 1

a i »

' ' -J ' vA t; it M/r, .> ..|

4
Introduction

Control systems for complex man i p u lator u tasks require precise- information of errors in position, orientation, and velocity

of the manipulator hand with respect to the object to be manipulated. This is particularly true in the final approach trajectory just prior to contact between the hand and object. In order

to provide this information in a form usable by a human operator

or a computer control system (or both) the National Engineering

Laboratory of the National Bureau of Standards has developed for

Marshall Space Flight Center a Proximity-Vision System suitably

configured to mount on the MSFC Protoflight Manipulator Arm

(PFMA). This Proximity-Vision System is a breadboard system

built in order to demonstrate the capabilities and analyze the

problems associated with using such a system in a real-time control mode. This report describes the theory of operation and the

physical and electrical components of the breadboard hardware

delivered to Marshall Space Flight Center. Performance testing

of this hardware is to be done by the recipient.

Technical Approach

A block diagram of the NBS Proximity-Vision System is shown

in Figure 1. This system consists of two separate but complementary subsystems.

1. A solid-state TV camera with 128 x 128 resolution elements is mounted on the wrist of the PFMA just behind the

fingers, as illustrated in Figure 2. This camera is positioned

so that its 36 degree field of view covers the manipulator

fingers and a region extending one meter in front of the fingers,

A photograph of the camera is shown in Figure 3.

Coordinated with this camera is a high intensity strobe

flash system with optics which projects a thin fan-shaped plane

of light into the region viewed by the camera in front of the

PFMA fingers. A photograph of the flash system is shown in Figure 4. Any diffusely reflecting object within the region in

front of the fingers reflects the light beam. The apparent position in the camera field of view of the reflected light makes it

possible to compute the distance of the reflecting object by simple tr iang u la t i on. The offset between the camera and the plane

of projected light d etermineSth e relationships between the distance and the apparent position of the reflected light.

The fact that the projected light is a plane means that the

intersection of the plane with a solid object will produce a reflected line of light as shown in Figure 5. The position of this

line indicates distance. Its shape conveys information concerning the depth profile of the object being illuminated, and its

end-points indicate the position and spacing of the edges to be

grasped. This is illustrated in Figure 6 and 7.

-3

-
2. Ihe second subsystem of the NBS Proximity-Vision System

consists of a pair of close-range infra-red proximity sensors imbedded in the tips of a pair of fingers suitable for mounting on

the PFMA. A photograph of the fingers is shown in Figure 8. Two

light emitting diodes/ one on each finger* project chopped

infra-red light into the region immediately in front of the

fingers. Two photo-transistors* also one on each finger* collect

light reflected from any diffusely reflecting object immediately

in front of or between the fingers. This is illustrated in Figure 9. The modulated light is synchronously detected in order to

discriminate against background illumination.

These short-range detectors are sensitive in the range of O

to 20 centimeters and measure only the strength of the reflected

light. Since the albedo of the reflecting object influences the

strength of the reflected infra-red signal* this subsystem

provides only relative position information unless the signals

are calibrated in the region when both the solid-state cameraflash system and the short range infra-red system overlap.

These short range detectors provide a measure of redundancy

to the system* particularly in the last few centimeters before

contact. They ^Iso provide centering information for large objects which overflow the field of view of the camera-flash system

at very close range.

Display-Control Console

Also included in the Proximity-Vision System is an operator

display-control console. A TV viewing monitor displays the video

signal from the solid state TV camera. Brightness controls on

the monitor and push buttons for remote adjustment of the "f"

stop on the camera enable the operator to optimize picture quality. The strobe flash is syncronized with the camera frame rate

so that the reflected line of light appears as a pulsing cursor

superimposed on the image. There is a selector switch which enables the operator to choose the pulse rate over the range from 3

per second to one every 1. 5 seconds.

Data from the infra-red system is displayed on the same monitor as the 128 x 128 camera data. Intensity of reflected light

from the two sensors is displayed as lengths of two bright lines

running up the right and left sides of the camera display raster.

The length of each line is proportional to the intensity of the

signal detected by the respective sensor.

-4 -
Computer Interface

The display-control console also contains a computer interface which can be used to interface the camera-flash system with

a coniputer.

The video signal from the TV camera is converted to a series

of 8 bit values by an analog to digital converter in the control

console. Synchronization with the computer is provided via a data accepted pulse from the computer. When in the computer mode

(which is selected by a switch on the front panel of the control

console) the video clock is interrupted after each A/D conversion

until the computer reads the 8 bit word and sends a data accepted

pulse.
Conclusions and Recommendations

A full series of operational tests of the NBS ProximityVision system with the MSFC Protoflight Manipulator Arm remains

to be done under the auspices of MSFC personnel. However/ preliminary tests performed prior to and during delivery and installation of the system on the PFMA suggest several areas of further

work.

1. Picture Quality

a) The intensity of the flash should be controlled so

that it can be increased for large distances and decreased at

short range. Preferably this should be accomplished automatically. This would best be done with a microprocessor which measures

the received signal and adjusts the strobe intensity accordingly.

The strobe intensity is best controlled by varying the duration

of the flash.

b) The flash repetition rate should also be varied so

that it is inversely proportional to distance. Thus as the manipulator closes on an object and the intensity of the strobe decreases/ the rate of the flashing strobe would increase. This

could also be accomplished by a microprocessor.

c) The iris adjustment to optimize the picture quality

should also be placed under microprocessor control.

2. Slip Ring Requirements

The PFMA has a continuous wrist—roll capability such that the

electrical connections between the op era tor—d i sp lay control station and the manipulator wrist must pass through a set of slip

rings. The NBS Proximity-Vision system in its present breadboard

form has a total of 23 leads which cross the continuous wristroll interface. Because it is a breadboard only no serious attempt was made to minimize the number of leads or to operate

through the existing slip rings.

Future versions of the Proximity-Vision system will need to

address this Slip ring problem. It would be possible to reduce

the required number of slip rings to 17 by allowing several of

the subsystems to share power supply voltages. This would require additional filtering of supply voltages to prevent cross

-6-
talk. Additional savings in slip ring requirements might be made

by eliminating the End-of-Line and End-of-Frame sync signals and

by putting the video clock generator on the wrist. This would

require the design of additional circuitry to reconstruct these

sync signals from the composite video.

The absolute minimum number of slip ring connections are as

follows:

Camera

1. + 15

2. -15

3. GND

4. Video

5. Strobe

Flash

6. +300 volts (or +5 volts with 3 amps capacity)

7. GND

8. Trigger

IR Sensor s-Emi t ter

5

9. Output #1

10. Output #2

11. +5 volts

12. GND

Iris Adjust

13. Motor Winding +

14. Motor Wind ing —

3. Computer Analysis of Proximity-Vision Data

It is recommended that a continued development effort be undertaken to make possible the automatic extraction of range and

position data from the proximity vision system. This would make

possible computer assisted terminal guidance and control of the

manipulator in grasping objects* extracting modules* and performing various other tasks which will be required of the PFMA and

similar systems in the future. The advantages would be increased

precision* speed* and dexterity of terminal movements* reduced

operator fatigue* and improved reliability and safety.

-7-
As a first step toward this goal# NBS recommends:

a) A microcomputer system for automatic image quality

and flash intensity and rate control. It is recommended that

this be a separate microcomputer from the PDP-11/35 which will

control the arm because of the unique demands of the video interface. Although it would be possible to connect the

Proximity-Vision system directly to the arm control computer

through a suitably configured interface* this is not recommended

because of the potential software problems due to conflicting

timing requirements of the two real time control systems.

b> A hardware f rame-to-f rame compare and threshold

system so that the line of light produced by the flash can be automatically separated -from the background image. A block diagram

of such a system is shown in Figure 10.

c) Software for feature extraction so as to automatically measure target parameters such as:

o distance

o closure rate

o s i z e o orientation

o depth profile

A block diagram of the recommended computer analysis system

is shown in Figure 11.

-8-
FIGURES

FIGURE 1; A block diagram of Proximity Vision System.

FIGURE 2; Mounting configuration of the Proximity Vision System.

FIGURE 3: The 128 X 128 solid state TV camera with mounting

bracket and iris adjust mechanism.

FIGURE 4: The flash head which generates the plane of light.

FIGURE 5; Relationship between the camera field of viewi the

plane of light/ and a viewed object.

FIGURE 6: A range graticule which when placed over the viewing

monitor gives the distance to any point on a reflected line of

light. Scale in inches.

FIGURE 7: Illustrations of various shaped objects viewed through

the monitor. The reflected line of light is shown as a heavy

black line across each object.

FIGURE 8: PFMA finger tips containing the close-range infra-red

proximity sensors.

FIGURE 9: A drawing illustrating the relative position of the

light emitting diodes (LEDs) and proximity sensors.

FIGURE 10: Block diagram of a hardware f rame-t o-f rame compare

system for automatic extraction of the line of light from background illumination.

FIGURE 11: Block diagram of recommended system for automatic control of image quality and extraction of target features.

-9

-

£X

v:aX<

rioijkh
CAMERA E

rb ui0) >

uzpo

CO •H>>

CMW

ct: P)

r*'.

Px,

Mounting

configuration

of

the

Proximit
1

JO

r 3

FlflllRE

i:

The

128 X

128

Rolld

state

TV

camera

with

mounting

bracket

and

iris

adjust

mechanism.
r

FIGURE

4:

The

flash

head

which

generates

the

plane

of

light
CAMERA

FJCUKF, S:

Re1nti

onsti i

f<
Y

DISTANCE

IN

CENTIMETERS

PIXEL COLUMN

X DISTANCE IN CENTIMETERS

FIGURE 6, A range graticule which when placed over the viewing monitor gives

the distance to any point on a reflected line of light. The X and Y

distances are measured in the coordinate system of the fingers. The X

axis passes through the two finger tips and the Y axis is parallel to

the wrist axis. The slight tile in the figure is due to a misalignment

of the chip in the camera.

PIXEL

ROW
BOX

(FRONT VIEW)

(OBLIOUE VIEW)

OBJECT WITH

RAISED

SURFACE

OBJECT WITH

DEPRESSED

SURFACE

(FRONT VIEW) (OBLIOUE VIEW)

figure 7: Illustrations of various shaped objects viewed through the

monitor. The reflected line of light is shown as a heavey

black line across each object.
FIGURE

8,:

PFMA

finger

tips

containing

the

close-range

infra-red

proximity

senso
PHOTO

TRANSISTOR

FIGURE

9:A

drawing

illustrating

the

relative

position

of

the

light

emitting

diodes

(LEDs)

and

proximity

sensors.
CCD

FRAME

BUFFER co

COEo

4-1 3

COMO

CME

OJ

4-J CO

CO

CD • Fh C

CO

O

CX *H E M O CO 0 C

04 E E 3

CO rH M r-l <M -H 1O TO M C

CO 00

J-l <M

a

cO

Oi ^ M

COS

TOM C|_i CO x; 4-1 -C

CO CjO •H 4h r-H O

CM E O

COM 0) 00 C '

cO -H

•H I—I

TO

CU XI

O 4-) O

rM CM

P3

O o1—I w

C3M

Pc-i
FIGURE

11:

Block

diagram

of

recommended

system

for

automatic

control

of

image

quality

and
M BS-1 1 4A (REV. 7-73)

U .S. DEPT. OF COMM. BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO. NBSIR 78-1576

2. GovT Accession

No.

3. Recipient’s Accession No.

4. TITLE AND SUBTITLE

Final Report - Proximity-Vision System for Protoflight

5. Publication Date Jan. 1979

6. Performing Organization f ode

7. AUTHOR(S)

James S. Albus, Ph.D.

8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, D.C. 20234

10. Project/Task/Work Unit No.

11. Contract/Grant No.

12. Sponsoring Organization Name and Complete Address (Street, City, State, ZIP)

Marshall Space Flight Center

Huntsville, Alabama 35812

13. Type of Report & Period Covered

14. Sponsoring Agency ^^-cifle

15. SUPPLEMENTARY NOTES

16. ABSTRACT' (A 200-word or less factual summary of most significant information. If document includes a significant

bibliography or literature survey, mention it here.) The NBS Proximity-Vision System consists of two separate but complementary

subsystems: 1) A solid-state TV camera with 128 x 128 resolution elements mounted

on the manipulator wrist. Coordinated with this camera is a high intensity strobe

flash system with optics which project a thin fan-shaped plane of light into the

region viewed by the camera. 2) A pair of close-range infra-red proximity sensors mounted in the fingertips.

The system is built for a manually controlled teleoperator. However, the

design is also suitable for computer image analysis in the control of an autonomous robot manipulator.

17. KEY WORDS (six to twelve entries; alphabetical order; capitalize only the first letter of the first key word unless a proper name; separated by semicolons)

Depth image; proximity sensor; robot vi^on; teleoperator vision; wrist-mounted TV

18. A V A 1 1. A B I L 1 I V Unlimited

[

1' or Otficial Distribution. Do Not Release to NTIS

[y Order 1- rom Sup. of Doc., U.S. Government Printing Office

Vf ashington, D.C. 20402, SD Cat. No. Cl 3 L o rder 1' rom National Technical Information Service (NTIS)

Springfield, Virginia 22151

19. SECURITY CLASS 21. NO. OF PAGES

(THIS REPORT)

20

UNCL ASSIFIED

20. SECURITY CLASS 22. Price (THIS PAGE)

$4. 00

UNCLASSIFIED

USCON^M.DC 29042-P74
'