Transparent Cockpit: Visual Assistance System for Vehicle
Using Retro-reflective Projection TechnologyTakumi Yoshida

KenseiJo t

Hideaki Nii §Kouta Minamizawa
Susumu Tachi 1Naoki Kawakami ¶The University of TokyoABSTRACT

For safety and operability of drivers while operating a vehicle, itis very important to obtain a wide field of vision. However, thespace available for setting up windows is limited. Therefore, wepropose a transparent cockpit," in which the image of a blind spot
is displayed on the inner wall of the vehicle using a retro-reflective
projection technology. In this system, the internal components ofthe vehicle, such as the doors and floor, are virtually transparent,
and the blind spot is clearly visible, as observed from a window.
In this paper, we describe the implementation of a prototype of the
proposed system and demonstrate its effectiveness by an evaluation

CameraBlindSpot/

CameraBlSot

Figure 1: Conceptual drawing of our proposed transparent cockpit.

experiment.

Keywords: Augmented reality, display technology
Index Terms: H.5. 1 [Information Interfaces and Presentation]:
Multimedia Information Systems Artificial, augmented, and virtual realities

1 INTRODUCTION
Supporting technologies for vehicle operators have gained increasing importance with the technological progress of vehicles suchas cars, airplanes, and helicopters. In particular, many studies
have been conducted on visual assistance systems using cameras
and image processing techniques, such as the traffic sign detection
system[l, 2], blind spot monitor[5, 11], night view system[13], andHIR(Human-Oriented Information Restructuring) system[8, 14].
Moreover, the design and realization of augmented-reality-based
vehicular assistance systems are proposed in recent years [7, 12].
The purpose of this study is to develop a visual assistance system that improves the safety and operability of vehicles. While
operating a vehicle, it is very important to obtain a wide field ofvision. However, the space available for setting up more windows
is limited. Therefore, the vehicle has certain blind spots that may
lead to accidents. Some studies have reported techniques for capturing these blind spots with cameras. In these techniques, imagesof blind spots are displayed on an in-vehicle display. However, inthese techniques, some problems are encountered: first, the operator cannot pay attention to the surroundings when observing thedisplay; secondly, it is difficult to display complete information
about the surrounding environment due to size limitations of thein-vehicle display.
Therefore, we propose a "transparent cockpit", wherein the image of a blind spot is displayed on the inner wall of the vehicle*e-mail:takumi-yoshida@ipc.i.u-tokyo.acjp
te-mail:kenseijo@ipc.i.u-tokyo.acjp

te-mail:kouta_minamizawa@ipc.i.u-tokyo.acjp§e-mail:hideaki nii@ipc.i.u-tokyo.acjp

q[e-maii:kawakami@star.t.u-tokyo.acjp
11e-mail:tachi@ star.t.u-tokyo.ac.jp

IEEE Virtual Reality 2008
8-12 March, Reno, Nevada, USA
978-1-4244-1971 -5/08/$25.00 ©C2008 IEEEFigure 2: Implementation on a car:the door seems transparent.

using a retro-reflective projection technology (RPT) [3, 10]. Figure 1 shows the concept of our proposed system. In this system,
the internal components of the vehicles such as the doors and seat,are virtually transparent as shown in Figure 2. Therefore, the blindspot is clearly visible, as observed from a window. RPT has been
researched as a visual display method for augmented reality. Furthermore, a technique for optical camouflage using the RPT has
been proposed [4]. We aim to realize our proposed system by applying the RPT to the vehicle cockpit.
A considerable number of projector-based augmented reality
systems have been proposed[6, 15]. In such systems, the projection would have to take occur on a complex-shaped surface. Therefore, geometric compensation is necessary. On the other hand, anondistorted image can be displayed on a complex-shaped screen
by using the RPT. In addition, there is an advantage of not obstructing the usual view because the image is not presented on parts otherthan the retro-reflector.
In the conventional method of optical camouflage using the RPT,
it is necessary to estimate the camera-to-subject distance. The scale
and position of the subject are incorrectly displayed if the assumed
camera-to-subject distance is different from the actual distance.
However, blind spots at various distances comprise various typesof subjects such as a walker, a guardrail, and buildings. Therefore,
this is a critical problem for our proposed system. To overcome it,
we propose a method to correct the image by measuring the camerato-subject distance.
In this paper, we describe the features and principle of the transparent cockpit system. Then, we describe the implementation of
the prototype system. Finally, we demonstrate the effectiveness of185

Head Position_-[nHoJectr~ Subject

Virtual Screen

Input Images

AIiL~~~~~~~~~~~~~~~~~~~~I

ComputerRero-reflector

Output Image

2Cameras

HMP

Motion Sensor~~~~~~~~~~Operator I~

OperatorRetro-reflector

Figure 3: System configuration of transparent cockpit.
Render the ImageReal Environment)

Retro-reflector
Y Projector

Observer

KHalfMirror

Point of ViewI

L+:4ICamerau~~~~~~~~Projective TexWre\AlO I

Figure 5: Gap(Aol, A02) between the subject(Pl, P2) and the displayed image(P-,PI) caused by the position of the virtual screen.

3.2 Image Correction Algorithm Based on Camera-tosubject Distance1~~~~~~~~~~-----

Figure 4: Concept of algorithm for rendering the image seen from theoperator's view. A, B, and C correspond to A', B', and C', respectively.
proposed system by the evaluation experiment,

PROPOSED SYSTEM
Figure 3 shows the configuration of our pproposedd system. We pothe th veil to catr
.igued3aws
ropose.
caea ousdth bln spts The
internal components of the vehicle, such as the doors and floor arecovered with the retro-reflector. The operator switches on the headmounted projector (HMP). The process flow of this system is asfollows,

1. Cameras capture images of the blind spots.
2. They are loaded into a computer.
3. The operator's point of view and line of sight are measured
using a motion sensor.
4. The camera-to-subject distance is measured by stereo

matching.
5. The image as seen from the operator's point of view is rendered by the computer in realtime.
6. The image is projected onto the retro-reflector from theHMP.

Then, the image of the blind spot is displayed on the interior
internal components of the vehicle. Therefore, these components
are virtually transparent, and the operator can intuitively identify
the surroundings. Thus, the safety and operability of the vehicle
can be improved.
3 PRI NCIPLE
3.1 Rendering Algorithm
The algorithm for rendering the image seen from the operator's
point of view is described below. First, we create a virtual envi186

Pronment on the computer as shown in Figure 4. The scale of thevirtual environment is same as that of the real environment. Then,a virtual screen A' is set up in the virtual world. Next, the image
AB
captured by camera B is projected onto the screen from point B' us|
ing projective texture mapping [9]. The distance between B' and A'
Camera-to-subject Distanceto the camera-to-subject distance in the real environment.
EisThisequalEnviromnent )I
C Virtual-----------------~~~~~~~~~~Virtualfor eachcamera.
theThen, werepeated point
process
image seen
(C). Finally,
themeasureof view
position of theisoperator's~ Screen \| |Virtualfrom~C'|is rendered.
A 'C'~~~~~~~~~~~~Thus, we convert the images captured by the cameras into an
image as seen from the operator.

'ourA 072

As described in section 1, it is necessary to estimate the camera-tosubject distance in the conventional method of optical camouflage
using the RPT. Figure 5 shows the gap between the subject and thedisplayed image caused by the position of the virtual screen. Whenthe subject is at P, P' is the correct point that should be displayed.
However, if the virtual screen is shifted AL1 forward, the subject isdisplayed at P1'. It should be noted that AO, increases with AL1.
This is also applicable to the case of AL2.
Therefore, we propose a method for correcting the image bymeasuringalgritmetomsurect
the camera-to-subject distance.
distance. We use the basic stereo
matchingtothematching algorithm measure distance.
In order to display information about the blind spot, we mustreconstruct the 3D environment of the surroundings. However, ingeneral, it is still very difficult to achieve such a reconstruction in
real-time. Therefore, we only measure a distance in the direction of
the operator's line of sight.
The algorithm of our proposed method is as follows. We selecttwo cameras that capture the direction of the operator's line of sight.
Then, feature points in the left camera image are detected. The
epipolar lines in the right camera image are calculated. Next, welocate the corresponding points on these lines by pattern matching.
The points closer to the direction of the operator's line of sight areselected. Subsequently, we measure the distances to the points bytriangulation. Finally, the virtual screen is set up at the average
distance, as mentioned in the previous section.

4 IMPLEMENTATION
We have developed a cockpit model and implemented the prototype system for the evaluation experiment (Figure 6). The shapeof the cockpit is cylindrical, and its radius is 1 m in length. The
wall and floor in the cockpit are covered with a retro-reflector (Reflite, 8318). There are 4 IEEE-1394 cameras (Point Grey Research,FireflyMV) outside the cockpit. Further, we use a 6 DOF motion
sensor (Shooting Star Technology, ADL- 1) for sensing the operator's point of view. We have developed a HMP by using a LED

_ -_-3|3~~~~~~~~~~~~~~~~~~~~~~Hal Mirror
Distance~~~~~~~~~~~~~~~~~~~~~~~~Camera-to-Subjecte|~~~~~~~~~Figure
1|
8: Experimental arrangement for evaluating image correction.
_ - llLED Projector-t

Lens Unit a2i
Half MirrolrtFigure 7: HMP:Head-mounted Projector.

projector (TOSHIBA, TDP-FFl)(Figure 7). It displays information
only for the left eye. Table 1 lists the specifications of the prototypesystem.

2m

3

Fiue9Coprsnfthoeao'svwbtenwtotcoe-tion and with correction in the evaluation experiment.

Table 1: Specifications of the prototype system
Camera frame rate [fps]30
The criteria for evaluation were (a) scale ratio, (b) horizontal angle
ll80 |error, and (c) vertical angle error of the marker and displayed image.
|Camera horizontal angle of view [deg] |||Camera resolution [pixel]||640x480l
HMP frame rate [fps]60 |5.2
Experimental Resultsll
60|HMP horizontal angle of view [deg] ||9 shows the comparison of the operator's view between|HMP resolution [pixel]||800 x600 |Figure
without correction and with correction in the evaluation experiment.
In each image, the upper and lower halves show the real environment and retro-reflector on which the image is displayed. respecEVALUATION EXPERIMENT OF IMAGE CORRECTION
tively.
that whenthe markeris at a distancesof
1 m andThe
3 m,result
thereshowsiS a large
gap between
the marker
and the dis5.1 Experiment Description
played image without the correction. On the other hand, when theIn section 3.2, we proposed the method for image correction based
correction is applied, the displayed image comparatively fits theon the camera-to-subject distance. To show the effects of thismarker at any distance.
Figure 10 shows the data of the criteria for evaluation. The near 1
method, we conducted the evaluation experiment using the prototype system. In this experiment, we measured the gap between theis superior at graph (a). The near 0 is superior at graph (b) and graphsubject and the displayed image with and without correction.
(c). The data of our proposed method with correction is superior in
As shown in Figure 8, we set a marker before the two cameras.
every graph. These experimental results further demonstrate theThe camera-to-subject distance was varied from 1 to 3 m. The diseffectiveness of our proposed method.
tance between the HMP and the retro-refiector was 1 m. Then, wecompared the results obtained with and without correction:
5.3 Discussion

1. Without correction: The virtual screen is fixed at 2 m fromthe cameras.
2. With correction: The virtual screen and marker were shiftedby the same distance using our proposed method.

The gap in the image was improved by the proposed technique.
However, the error margin still remains. As a cause of the error
margin, it is thought that the coordinates of the virtual environment
do not accurately correspond to those of the real environment. To187

(a)

1.5

the brightness and the contrast of the image according to the situation.

1.4*

o1.4 ~137

1.2W,d 1vz 0.90.8[deg]

In this paper, we proposed the visual assistance system for vehicle
using RPT, named "transparent cockpit". This system enables theWith Correction+operator to observe the surroundings in the blind spot through theinternal components of the vehicle. We developed the prototype
system applying the image correction method based on the camerato-subject distance. The result showed that the gap between the disand real environment was improved by the proposal
played image12ositionmethod.
Finally, we implemented the limited
system on an actual
car and carried out a test run. Then we confirmed the effectiveness[deg]Ac)of our proposed sustem.
[deg]
(c)
There still exist some gaps between the subject in the real envi8
ronment and the displayed image. It is neccesary to improve the[1 6 Q- ° 1<= o ~~~~~0 u° 4 k \measurement accuracy of the camera-to-subject distance. In the fu2/ 4 \ture, we will implement the complete system in a vehicle to deter2
mine the improvement in the safety and operability of the vehicle.

Without Correction

10.7
0.6

CONCLUSION.

-°-X-,
a23[m]
Marker Position(b)
k J2~~~~~o.O

O_4 024 -2r,oREFERENCES

4O -6 O
1-4
23 [m]

1Marker Position

A. dela Escalera, L. Moreno, M. A. Salichs, and J. M. Amingol. Road
[1] traffic
sign detection and classification. IEEE Transaction Industrial

..

3 [i]

2Electronics, 44.

Marker Position[2] D. M. Gavrila. Traffic sign recognition revisited. In In Proceedings ofthe 21st DAGM Symposiumfor Mustererkennung, pages 86-93, 1999.

[3] M. Inami, N. Kawakami, D. Sekiguchi, Y Yanagida, T. Maeda,Figure 10: Results of the evaluation experiment.

E0and S. Tachi. Visuo-haptic display using head-mounted projector.
In Proceedings of IEEE Virtual Reality 2000, pages 233-240, New
Brunswick, NJ, March 2000.
[4] M. Inami, N. Kawakami, and S. Tachi. Optical camouflage using
retro-reflective projection technology. In Proceedings of the Second
IEEE and ACM International Symposium on Mixed and AugmentedReality (ISMAR 03), 2003.
[5] S. Okamoto, M. Nakagawa, K. Nobri, and A. Morimura. Developmentof parking assistance system ushg virtual viewpoint. In Proc. of 7thWorld Congress on Intelligent Transport Systems, 2000.
-Figure 11: Implementation of our proposed system on an actual car.
The lower part of a person(left) and traffic signs on the road(right)
are clearly visible, as seen from a window.

decrease the error margin, it is necessary to develop a calibration
method and improve the accuracy of the stereo matching.
One of the problems encountered in the proposed system is thatthe displayed image is distorted when the subject is close to thecameras because the distance of subject is at distance too close forrecognition. The area where the distance information can be acquired depends on the arrangement of the cameras. It is necessary
to decide the arrangement of the cameras according to the application such as to cars, airplanes, or helicopters.

[7] A. Rodriguez, M. Foglia, and J. P. Rolland. Embedded training display

technology for the army's future combat vehicles. In Proceedings ofthe 24th Army Science Conference, pages 1-6.

[8] H. Sakai, M. Koyamaishi, T. Fuji, and M. Tanimoto. Experiment ofsafety drive in an intersection by intersection by visual assistances
based on hir system. In IEEE Intelligent Vehicles Symposium, pages
265-270, 2003.
[9] M. Segal, C. Korobkin, R. van Widenfelt, J. Foran, and P. Haeberli.
Fast shadows and lighting effects using texture mapping. In In Proceedings ofSIGGRAPH '92, pages 249-252, 1992.
[10] S. Tachi. Augmented telexistence. Mixed Reality Merging Real andVirtual Worlds, pages 251-260, 1999.
[11] Y Tanaka, T. Kakinami, Y. lwata, M. Nhmura, S. Hiramatsu, andA. Hibi. Image projection based parking assist system. In Proc. ofIEEE Intelligent Vehicle Symposium 2001,, pages 399-404, 2001.

66
TESTTEST RUN
RUN

We carried out a test run in order to demonstrate the effectivenessof our proposed sustem. We implemented the basic system on an
actual car. We applied the retro-reflector to the passenger's door anddashboard. In this experiment, we used only one camera, and did
not perform the image correction based on the camera-to-subject
ditace The poito of th HM
distane.
TheraposiinofteHPra.ixdfied
waitwsntwr

anniby the operator.
Figure 11 shows the experimental result. As shown in Figure
11, the internal components of the car are virtually transparent, andthe location of the blind spot is clearly visible. This result qualitatively demonstrates the effectiveness of our proposed system. The
displayed image was bright enough in the daytime. But if it is dark
Outside, the displayed image darkens. So it is neccesary to adjust

188[6] R. Raskar, M. Cutts, G. Welch, and W. Stuerzlinger. Efficient image
generation for multiprojector and multisurface displays. In RenderingTechniques '98, Springer Verlag, pages 139-144, 1998.

[12] M. Tonnis and G. Klinker. Effective control of a car driver * sattention
for visual and acoustic guidance towards the direction of imminent
dangers. In The Fifth IEEE and ACM International Symposium on
Mixed2006. and Augmented Reality, Santa Barbera, CA, USA, Oct. 22 25,K.
[13] K. Toyofuku, Y. lwata, Y. Hagisato, and T. Kumasaka. The "nightview system" using near-infrared light. In SAE 2003 World Congress& Exhibition, 2003.
[14] K. Toyota, T Fuji, T. Kimoto, and M. Tanimoto. A proposal of hir
system for its. In IEEE Intelligent Vehicle Symposium, pages 540544, 2000.
[15] G. Wetzstein and 0. Bimber. Radiometric compensation through inverse light transport. In Pacific Graphics, 2007.