US006731331B1(12) United States Patent(10) Patent N0.:
US 6,731,331 B1(45) Date of Patent:
May 4, 2004

Watabe et al.

(5 6)(54) REMOTE-CONTROLLED SHOOTING

References Cited

SYSTEM, VIDEO CAMERA APPARATUS
AND REMOTE-CONTROLLED SHOOTINGU.S. PATENT DOCUMENTS

METHOD
5,844,602 A
6,130,705 A(75) Inventors: Shigehiko Watabe, Tokyo (JP); Tetsuji
Naoi, Tokyo (JP); Ushio Inoue, Tokyo(JP); Kouji Miyajima, Tokyo (JP);* 12/1998* 10/2000

Lareau et a1. ............. .. 348/144
Lareau et a1. ............. .. 348/144

FOREIGN PATENT DOCUMENTS

Hatsuo Kimura, Tokyo (JP);JPJP

Toshitsugu Katsumata, Tokyo (JP);Satoshi Nishiyama, Tokyo (JP); YukioTanaka, Tokyo (JP); Akinori Kosako,Tokyo (JP); Yoshikazu Osamemoto,Primary Examiner—Vu Le(73) Assignees: Mitsubishi Denki Kabushiki Kaisha,Tokyo (JP); NTT Data Corporation,

Assistant Examiner—BehrooZ Sen?

(74) Attorney, Agent, or Firm—Burns, Doane, SWecker &Mathis, LLP

Tokyo (JP)Notice:9/1998
9/1998* cited by examiner

Tokyo (JP); Hiroyoshi Ueno, Tokyo(JP); Akio Komiya, Tokyo (JP)(*)

10-257474
10-257483

Subject to any disclaimer, the term of thispatent is extended or adjusted under 35

USC 154(b) by 727 days.

(57)

ABSTRACT

A shooting instruction device 1 selects one of preset shoot(21) Appl. No.: 09/610,610

ing patterns 101 according to the type and scale of thedisaster to cope With, then inputs shooting conditions 112
corresponding to the selected shooting pattern, and sends a(22) Filed:shooting instruction 103. Avideo camera apparatus 2 loaded(30)

on a helicopter 3 makes a ?ight plan 106 based on theForeign Application Priority Data

Jul. 7, 1999

(51)

Jul. 5, 2000

received shooting instruction 103, then shoots the position(JP) ......................................... .. 11-193440

Int. Cl.7 ................................................ .. H04N 7/18

(52)

US.

(58)

Field of Search ............................... .. 348/140, 141,Cl.

. . . . . . . . . . . . . . . .

. . . . . . . . . . ..

348/144of a shooting target, then adds a still picture mark to a video

signal 111, and sends it together With associated data 114. An
information display device 4 extracts a still picture from avideo signal 113 With the still picture mark, and displays thestill picture together With the associated data 114.

348/142, 143, 144, 147, 254, 123, 220_225,
160, 145, 146, 148

25 Claims, 20 Drawing Sheets

VIDEO CAMERAAPPARATUS

4sl~ SHOOTINGINSTRUCTION

DEVICE

BSI/JILCPEYU.S. Patent

May 4, 2004

Sheet 1 0f 20

US 6,731,331 B1

FIG. 1S3
VIDEO CAMERAAPPARATUS

TF5,’
2

4
$

1“ SHOOTINGINSTRUCTION
DEVICE(P) BASIC

55212:‘?

(P1) SINGLE-POINT SHOOTINGSHOOTING

PATTERN(P2) MULTIPLE- POINT SHOOTING(P3) LINE SHOOTING(P4) AREA SHOOTING(P5) SPOTLIGHT SHOOTING(P6) PANORAMIC SHOOTING(P7) FREE SHOOTING(Q) MODELSHOOTINGPATTERN(Q1) MODEL SHOOTING PATTERN(Q2) MODEL SHOOTING PATTERNU.S. Patent

May 4, 2004

US 6,731,331 B1

Sheet 2 0f 20

FIG.2
2325104
1MAP DATA<‘ DISPLAY

STORAGE PART

106“

2124;

I‘

2(6L FLIGHT PATH

7 CONTROL PART

104~SHOOTINGINSTRUCTION

107

FLIGHT PATHDISPLAY PART127 10828II’ SHOOTING"g, CAMERACONTROL PART(_I_J110

SHOOTING~ PLANNING

RECEIVING PART 2 PART\103\"105

POSlTIONéATTITUDI

109111~N

DETECTING

105PART»/112

23PART~10?’29v
SHOOTING

105 Tv

I; INFORMATION

166

GENERATING PART
1 13 ~‘y~ 1 14

“ SHOOTING RESULTS

VIDEO CAMERA APPARATUS

IR‘ ‘I 1 SMITTING PART22

II3~~1I4\V

41~ SHOOTING RESULTS
SHOOTING

RECEIVING PARTINSTRUCTION~ 14~

TRANSMITTING PART

42

L103
SHOOTING'

EXTRACTING PART103

INSTRUCTION INPUTPART115 L‘

101~ lsl T—“ 12MAP DATA

H14v
I
43 ~ DATABASE STORAGE

102

SHOOTINGv 1 14I STILL PICTUREr’ 13I~113PART44‘

_6_‘’
11

MAP DATAPATTERN

STORAGESTORAGESTORAGE PARTPARTPARTS(115‘ 114

SHOOTING, INFORMATIONDISPLAY PART)SHOOTING INSTRUCTION DEVICEDISPLAY DEVICEss

1445

U.S. Patent

May 4, 2004

Sheet 3 0f 20

US 6,731,331 B1

FIG.4

A: SHOOTING TARGET
B: SHOOTING DISTANCE

E: STANDBY AREA

FIG.5

E

B

A: SHOOTING TARGET
B: SHOOTING DISTANCE
E: STANDBY AREAmU.S. Patent

May 4, 2004

Sheet 4 0f 20

US 6,731,331 B1

FIG.6

A

WPT: WAY POINT
D: STILL PICTURE MARKING POSITION
E: STANDBY AREA

FIG.7
WPT(12)

D66)\WPT(1)13(1)D61) WPT(11)/II

D(2) 13(3)D(6)

WPT: WAY POINT
D: STILL PICTURE MARKING POSITIONU.S. Patent

May 4, 2004

Sheet 5 0f 20

FIG.8

D06) 13(2)D(1) 13(3)A: SHOOTING TARGET
D: STILL PICTURE MARKING POSITION

A: SHOOTING TARGET

US 6,731,331 B1

U.S. Patent

May 4, 2004

US 6,731,331 B1

Sheet 6 0f 20

FIG.1()SHOOTING
AREA
A: SHOOTING TARGET~1O3"5}MODEL

SHOOTING INSTRUCTION

gliggggf ~ 51

DECISION PARTSTORAGE‘f 103PART

SHOOTING INSTRUCTION N 53

STORAGE PARTV~ 103194'FLIGHT PLAN

2 54

PREPARING PART‘195‘~106
FLIGHT PLAN

N 55

STORAGE PART~l06110I‘FLIGHT PLANN 56
MANAGEMENT PART .

SHOOTING

PLANNING PARTl~106V

24

U.S. Patent

May 4, 2004

Sheet 7 0f 20

US 6,731,331 B1( START )V
SELECT SHOOTING PATTERN~ STl\INPUT SHOOTING CONDITIONSCORRESPONDING TO SELECTED SHOOTING~ ST2PATTERNVOUTPUT SHOOTING INSTRUCTION BASED ONSELECTED SHOOTING PATTERN AND INPUTSHOOTING CONDITIONS OR ON SELECTEDSHOOTING PATTERNN ST3\SEND SHOOTING INSTRUCTION~ ST4END%W%f’%/4/’
izé%///2’//T~ MAPW
?J///¢-'\/ AREA
SH T TO BE

U.S. Patent

May 4, 2004

Sheet 8 0f 20

US 6,731,331 B1

FIG.14ITEM

CONTENTS

INSTRUCTION NO.

INSTRUCTION ID NO.

DATEDATE OF SENDING INSTRUCTION

TIMETIME OF SENDING INSTRUCTION

TYPE

SHOOTING PATTERNPOSITION OF TARGETLATITUDE & LONGITUDE

ALTITUDE FORSHOOTING

ALTITUDE OF HELICOPTER DURING
SHOOTING

PARAMETERS OFVISIBLE-LIGHT

USE/NONUSE OF CAMERAMAGNIFICATIONCAMERAFHSTERPARAMETERS OF
INFRARED CAMERAGAINUSE/NONUSE OF CAMERAMAGNIFICATIONFILTERGAIN

STILL PICTURE SIZEACTUAL WIDTH OF STILL PICTUREMESSAGEMESSAGE TO BE DISPLAYED ON FLIGHT
PATH DISPLAY PART ON HELICOPTERSTILL PICTURE
PROCESSINGSPECIFY PROCESSING METHOD(COMBINING STILL PICTURES, ESTIMATIONOF BURNING AREA)

U.S. Patent

May 4, 2004

Sheet 9 0f 20

US 6,731,331 B1FIG. 15IRECEIVE SHOOTING INSTRUCTION~ STIIFORMULATE FLIGHT PLAN BY REFERENCE TO MAPDATA AND CURRENT POSITION/ATTITUDE~ ST12INFORMATION OF HELICOPTER BASED ON SHOOTINGINSTRUCTIONIDISPLAY FLIGHT PATH OF HELICOPTER BASED ONN ST13FLIGHT PLAN AND MAP DATA'vOUTPUT CAMERA CONTROL INFORMATION BASED ONFLIGHT PLAN AND CURRENT POSITION/ATTITUDE~ STI4INFORMATION OF HELICOPTERISHOOT TARGET BASED ON CAMERA CONTROLINFORMATION AND OUTPUT VIDEO SIGNALS ANDSHOOTING CONDITIONS~ STISIOUTPUT STILL PICTURE MARKING INSTRUCTIONBASED ON FLIGHT PLAN AND CURRENT POSITION/~ STI6ATTITUDE INFORMATION OF HELICOPTERVPUT STHJL PICTURE MARK TO VIDEO SIGNALS, OUTPUT
VIDEO SIGNALS WITH STILL PICTURE MARK, AND OUTPUT N ST17POSITION/ATTITUDE INFORMATION AND SHOOTINGCONDITIONS AT THE TIIVIE STILL PICTURE MARK WAS PUTISEND VIDEO SIGNAL WITH STILL PICTURE MARK AND N 5T1 8

ASSOCIATED DATA

LIGHT PLAN END?

OUTPUT PROGRESS-OF-SHOOTING INFORMATION

STZO

U.S. Patent

May 4, 2004

Sheet 10 0f 20

US 6,731,331 B1\ WPT(6)WPT(5)

WPT(3)+ 4

FLIGHT

PATHD6)
WPTQ)

I,13(2)
1'WPW)
WPT( I)

I,__ '1-___..-.(-__---.I _ 11-’!--'I,<1)
LiI/I SHOOTING AREA/Z 7/// HELICOIETER(CURRENT POSITION)

FIG . 1 7
CHARACTER
INFORMATIONSHOOTING
AREA

MESSAGE * * * * *FLIGHT PATH
SPEED *** Km

HEIGHT *** m

FLYING FLIGHTSPEED
ALTITUDE M AP DISPLAY
HELICOPTER(CURRENT POSITION)

U.S. Patent

May 4, 2004

Sheet 11 0f 20

US 6,731,331 B1

FIG. 18

F1G1
F2
G2
F3r, G3

3/////////////////////////////A

F4STILL} PICTURE
MARK

WWW/A G4

F60

G60
F1, F2,- - - ', F60:NTSC FIELD
B1, B2; ' ' -, B6O2VERTICAL SYNC SIGNALU.S. PatentMay 4, 2004

Sheet 12 0f 20

US 6,731,331 B1FIG. 19
ITEM

CONTENTS

INSTRUCTION NO.

INSTRUCTION ID NO.

DATEDATE OF SHOOTINGTIMETIME OF SHOOTINGTYPE

SHOOTING PATTERNPOSITION OF

LATITUDE/LONGITUDE OF HELICOPTER

SHOOTING

DURING SHOOTINGALTITUDE

ALTITUDE OF HELICOPTER DURING
SHOOTING

STILL PICTURE SIZEACTUAL WIDTH OF STILL PICTURE

NUMBER OF STILLPICTURESTOTAL NUMBER OF STILL PICTURES

NO. OF STILLPICTURES

NO. OF EACH STILL PICTURE

POSITION OF
SHOOTING TARGETLATITUDE/LONGITUDE OF THE CENTER OF
EACH STILL PICTURE

ALTITUDE OF

ALTITUDE OF THE CENTER OF EACH STILLSHOOTING TARGETPICTURE

AZIMUTH OFAZIMUTH ANGLECAMERAANGLE OF DEPRESSIONANGLE OF ROTATIONPARAMETERS OF

USE/NONUSE‘VISIBLE-LIGHTCAMERAMAGNIFICATIONFILTERGAIN
USE/NONUSEMAGNIFICATIONFILTERGAIN

PARAMETERS OF
INFRAREDCAMERAU.S. Patent

May 4, 2004

Sheet 13 0f 20US 6,731,331 B1FIGZOI START )VRECEIVE SHOOTING INSTRUCTION~ ST21ST22MODELSHOOTINGPATTERN?

ST23SREAD OUT BASIC SHOOTING PATTERN ANDSHOOTING CONDITIONS FROM MODELSHOOTING PATTERN STORAGE PART4I

STORE SHOOTING INSTRUCTION~ ST24VREAD OUT SHOOTING INSTRUCTION, AND
FORM FLIGHT PLAN BY REFERENCE TO MAP ~ ST25DATA (ALTITUDE DATA) AND POSITION/ALTITUDE INFORMATION OF HELICOPTERISTORE FLIGHT PLAN~ ST26V

READ OUT AND OUTPUT FLIGHT PLAN ~ ST27END

U.S. PatentMay 4, 2004

Sheet 14 0f 20

US 6,731,331 B1FIG.21( START )VRECEIVE VIDEO SIGNAL WITH STILL PICTUREMARK AND ASSOCIATED DATA,, ST31VEXTRACT STILL PICTURE FROM VIDEO
SIGNAL WITH STILL PICTURE MARK, AND
STORE IT TOGETHER WITH ASSOCIATED DATA~ ST32VREAD OUT STILL PICTURE AND ASSOCIATED DATA w 3T33I

DISPLAY STILL PICTURE AND ASSOCIATED DATA ,._, ST34
ON MAP DATA

END

FIG.22

P////%WM“I"

VAURNING AREA~ MAP DISPLAY

M’ V% ’\’ SHOOTINGRANGE

U.S. PatentMay 4, 2004

Sheet 15 0f 20

US 6,731,331 B1FIG . 23ADc

CURRENT POSITIONEBDMOUNTAIN M A
A: POSITION OF SHOOTING TARGET
B: SHOOTIGN DISTANCE

C: APPROACH PATH
D: STILL PICTURE MARKING POSITION
E: STANDBY AREA

FIG.26
WPT(2)D(4)WPT(4)

WPT(3)CURRENTPOSITIONU.S. Patent

May 4, 2004

Sheet 16 0f 20

US 6,731,331 B1

FIG.24I START I
VOBTAIN ALTITUDE DATA OF POSITION A OF0, ST41

SHOOTING TARGETNOBTAIN SHOOTING DISTANCE FROM POSITION N ST42A TO OBTAIN SPECIFIED STILL PICTURE SIZEV

DETECT CURRENT POSITION OF HELICOPTER~ $T43

JOIN CURRENT POSITION OF HELICOPTER AND N ST44POSITION A TO DEFINE APPROACH PATH ClOBTAIN ALTITUDE DATA FROM POSITION ATO B ON PATH C

N ST45SET, AS STILL PICTURE MARKING POSITION D,INTERSECTION OF LOWEST ALTITUDE AND
SHOOTING DISTANCE B ON PATH C FROM,, ST46

ALTITUDE DATA FROM POSITION A TO BVCALCULATE FROM SPECIFIED STILL PICTURESIZE FLYING SPEED V TO PROVIDE NON-»/ ST47

BLURRED IMAGESET, AS STANDBY AIRSPACE E, POSITION
B+V><30 SEC FROM POSITION A@,, 5T43

U.S. PatentMay 4, 2004Sheet 18 0f 20US 6,731,331 B1FIG.27( START )VNUMBER PLURAL SPECIFIED WAY POINTS(WPT) IN ORDER OF PASSAGEVCOMPUTE FROM SPECIFIED STILL PICTURE
SIZE FLYING SPEED V TO PROVIDE SHARPSTILL PICTURESN STSIM ST52VSET POSITION WPT(1)+V><3O SEC AS STANDBYAREAA, ST53VSET WPT(O) AT INTERSECTION OF LINE JOINING
CURRENT POSITION AND WPT(I) AND STANDBY ~ ST54AREA E'V

DETERMINE STILL PICTURE MARKINGPOSITIONS D(n) so THAT STILL PICTURESOvERLAP ABOUT 20% AT THE TIME OFEXTRACTING SPECIFIED STILL PICTURE SIZEN ST55

' BETWEEN WPT(n) AND WPT(n+l)
V

OBTAIN ALTITUDE DATA ON D(n)N ST56VDETERMINE FLIGHT ALTITUDE BASED ON

MAXIMUM ALTITUDE BETWEEN WPT(n)AND WPT(n+l)

NO

LAST WAY POINT?

~ ST57

U.S. Patent

May 4, 2004

02 0Sheet 19 0f 20

US 6,731,331 B1>0270;2>02 0>203020$20 “20 02"05:20 02 0 M0 20m 02m0520 02 0 m0 20 052020

00R0@272N02

002 02 0

20 0 202600000720 0002 m00 2002020

2@00700 M002 0n00267 30

20002020 720__"

0ZUOF<CHL>:EWIm
20@0