倉田奨励金研究報告書 第 52 集
2021 年度（第 53 回）助成

自然科学・工学－都市・交通：1 年助成

奨励金№1496

高速視線焦点制御と高速画像処理に基づく広域高精細車載
ビジョンシステム
平野正浩

東京大学

助教

Onboard High-speed Vision System for Broad andHigh-definition Observation
Masahiro Hirano,
The University of Tokyo, Assistant Professor

本研究では、高速カメラに取り付けた電動ズームレンズと 2 軸ガルバノミラーを用いた新しいアクティブビジョンシステムである Falcon を提案する。このシステムは、近くから遠くまで広域かつ高精細なイメージングを可能とする。Falcon を駆動するために、マッピングベースの高精度キャリブレーション手法と効率的なビジュアルフィードバック手法を提案し、カーブミラーのトラッキングを実現することで提案システムの有効性を示した。
We propose a novel active vision system, called Falcon, that includes an electric zoom lens attached to a highspeed camera with a pair of galvanometer mirrors. This system enables high-resolution imagery of an object moving
over a wide range, from near to far.
We also propose a new mapping-based calibration method and present a robust and lightweight visual feedback
method for object tracking. We constructed a prototype and conducted tracking experiments in an indoor
environment, which showed the superiority of our method. Additionally, we successfully demonstrated continuousand high-resolution imaging of a curved mirror on public roads while the vehicle was in motion.

1．研究内容between the measurement area and resolution.

1. Introduction

However, there is a contrast in importance inSensing the surrounding environment is adifferent parts of the surrounding environment.

fundamental function of robots, includingThis suggests that by optically adjusting theautonomous vehicles. While recent advances inmeasurement area according to the context of theelectronics have enabled the development ofsurrounding environment, areas important forvarious sensors such as LiDAR and millimeter

driving can be obser ved with high resolution,

radar, vision sensors play an irreplaceable role due

contributing to safer and more efficient automaticto their high capability in environmental

driving. For example, by focusing on a cur ved

recognition, which is widely appreciated in many

mirror, the system can observe a blind area aroundvehicle applications.

the corner in detail.

Conventional onboard vision sensors observe theIn this study, we introduce a novel active

surrounding scene homogeneously with a fixedonboard vision system called Falcon, which

optical system, which inherently has a tradeoff

overcomes the limitations of conventional systems倉田奨励金研究報告書 第 52 集
2021 年度（第 53 回）助成

自然科学・工学－都市・交通：1 年助成

by using a 2-axis galvano-mirror and variable optics

camera (internal camera), as shown in Fig. 2. The

with adjustable zoom and focal position（Fig. 1）.

optical axis of the internal camera was carefully

This enables the continuous observation of a widealigned to hit the center of the first mirror and isand distant area at high resolution, enabling safer

parallel to the axis of the second mirror. Theand more efficient automatic driving. To ensureinternal camera observes the foreground scene byaccurate and reliable performance, we developed achanging its gaze direction through a pair of

tracking mechanism that let Falcon to keep thereflections from the fast-tilting mirrors. In addition

object at the center of the field of view, even whento the internal camera, two wide-angle video rate

the objectʼs relative position changes rapidly owing

cameras (external cameras) were mounted on theto vehicle movement or the vibration. We also

Falcon system. These cameras obser ve thepropose a novel mapping-based calibration method

foreground scene to locate areas in environmentsthat ensures the accuracy and precision of thewhere close observations are required.

Falconʼs performance, as well as a visual feedback
method that utilizes the calibration data to optimize
object tracking. We developed a prototype andsuccessfully demonstrated the continuous tracking
of a curved mirror as an example of an area in thesurrounding environment while the ego vehicle
was in motion.

Fig. 2 Schematic drawing of Falcon.

2.2. Calibration
2.2.1. Model-based calibration
We calibrated the Falcon system using an opticalmodel originally developed for a Saccade Mirror,
which also comprises a pair of galvanometer
mirrors. During the model-based calibration
Fig. 1 Falcon enables high-resolution imagery of amoving object in a wide range from near to far.

process, the zoom ratio (RZ) was set to be constant.
Prior to calibrating the Falcon, we first performed
stereo calibration of the two wide-angle cameras.

2. Falcon: A Wide-and-deep Onboard Active

To achieve this, we placed a large calibration board

Vision Systemin front of the system and made the Falcon scan

2.1. Components

the board to record the mirror angles, along withThe Falcon system comprises a dual-axis

the recognized corner pointʼs 3D position

galvanometer mir ror and an electrically(determined through triangulation) and its

controllable zoom lens equipped with a high-speed

projected point onto the internal camera. We then倉田奨励金研究報告書 第 52 集
2021 年度（第 53 回）助成

自然科学・工学－都市・交通：1 年助成

2.3. Control

calculated the calibration parameters throughoptimization using the model and measurements.

An active vision system commonly employs aUsing these parameters, we obtain a model-based

PID controller to adjust the mirrorʼs angle toprojection

minimize the distance to the center. However,  P; R Z adjusting the control parameters can lead towhich maps a point P on a reference plane ΠR to a

mapping-based visual feedback method for active

point p on an image plane ΠI, where ΠR is a planevision systems that maximize the use of accurate

parallel to Px－Py plane, that is, Pz = 1. However, thecalibration. Let p t   I be the location of the target

accuracy of the model-based calibration was

at time T where the mirror angles are α, β, and let

insufficient chiefly because of modeling errors.

pc   I be the center of the image.

instability. In this work, we propose a novel

New mirror angles  ,   are required to drive
2.2.2. Mapping-based calibration

the mirrors to capture the object at the center,To further improve calibration accuracy, we

which is equivalent to deriving the location of thepropose a method to refine the projection throughobject on the reference plane P t   1  ,   .

an additional calibration process. First, we captured

However, we cannot directly obtain P t from p t witha pair of images of the calibration board using

μ–1 because μ describes the relationship between

wide-angle cameras to obtain the 3D positions

the mirror angles and a point on the reference Px , Py , Pz  of the corner points of the board,plane, which is always mapped to the center of thewhich were placed in front of the system.

image.

For each corner point, we set the mirror angles

To address this issue, we assumed the accuracy

α (pan) and β (tilt) to obser ve the corner pointof the model-based calibration. It is assumed thatat the center using parameters from the model-

the distances between points  1  ,    ,  1  ,  based calibration and we adjusted the angles

are approximately equal to the distances between

delicately so that the center of the image coincided

points  1 p t ; R Z ,  1 p c ; R Z .

with the corner point. We then obtained a refinedmapping M : Px , Py   ,  , whereyxzy P , P    P / P , P / P xy P , P    P / P , P / P  is the projected corner point onx

zx

 1  ,     1  ,     1 p t ; R Z   1 p c ; R Zzyz



where R Z is the zoom ratio at time T, which is

the reference plane ΠR. This mapping accurately

updated in real time. The approximate mirror

corresponded to the gaze direction relative toangles are obtained as follows:the angles of the mirror. After repeating thisprocess several times to obtain a collection of ,      P t mappings Mi, we approximated the mappings Mi  Pc  P t  Pcby a pair of continuous functions μα and μβ using   1  ,     1  ,      1  ,  cubic polynomial surfaces with bisquare robust   1  ,     1 p t ; R Z   1 p c ; R Zestimators.

 ,      Px , Py      Px , Py  ,   Px , Py  

We refer to this as mapping-based feedbackcontrol. Note that we approximate the inverse of μ倉田奨励金研究報告書 第 52 集
2021 年度（第 53 回）助成

自然科学・工学－都市・交通：1 年助成

in the same manner as μ because μ is injective.

roof 1.85 m above the ground as shown in Fig. 1.

Because all the functions are algebraic and pre-The zoom ratio was initially set to 3.0, and thecomputable, the computational cost is low. In

focus position was set to 30.0 m.

addition, unlike PID control, no parameters need tobe tuned by the users, which is advantageous for3.2. Results
Fig. 4 illustrates how Falcon was able to observe

practical use.

a cur ved mirror. The cur ved mirror was stably
3. Curved mirror tracking

tracked at the center of the image, thanks to the

3.1. Scenario

visual feedback provided by the proposed method.

We demonstrated the use of Falcon for cur ved

We compared the images captured by the proposed

mirror tracking on public roads as an application ofsystem with those captured by an external camera,

intelligent vehicles. Curved mirrors are commonly

as shown in Fig. 5. We confirmed that the situation

used in Japan to help drivers observe blind spots inbehind the corner was clearly visible on the image

their vehicles. To conduct tests, we firmly mounted

reflected by the cur ved mirror. Even though thea prototype (Fig. 3) on top of the passenger vehicle

external camera had a ver y high resolution

External camera

Fig. 3 A prototype of Falcon

FalconFig. 5 Comparison of the images of the curvedmirror.

Fig. 4 Curved mirror tracking倉田奨励金研究報告書 第 52 集
2021 年度（第 53 回）助成

(2880×1860), we obser ved that the imager y by
Falcon was superior in terms of image resolution,
which contributed to improving subsequent
processes.
4. Conclusion
In this study, we propose an innovative active
vision system composed of an electric zoom lens
attached to a high-speed camera with a pair of fasttilting mirrors for gaze control. Furthermore, weintroduce a novel and accurate mapping-based
calibration method for Falcon in conjunction withexternal cameras. We also proposed a mappingbased feedback control method for robust object
tracking with minimal computational overhead. We
demonstrated the systemʼs efficacy by tracking acur ved mirror on public roads and show that
Falcon produced better quality imager y than acommercial high-resolution onboard camera.
Future work includes the development of practical
ITS applications using Falcon, such as traffic signal
recognition and dangerous spot observations.

自然科学・工学－都市・交通：1 年助成