Transparent Cockpit
Takumi Yoshida∗

Kensei Jo†

Kouta Minamizawa‡
Susumu Tachik
The University of Tokyo

Hideaki Nii§

Naoki Kawakami¶

Figure 1: Image of Usual Cockpit (left) and the Transparent Cockpit (right).

1 Executive Summary
We propose a “transparent cockpit” that is aimed toward improving
the operativeness, safety, and comfort during controlling a vehicle.
In the transparent cockpit, the interior appointments of the vehicle
are virtually transparent; this is enabled by using the retro-reflective
projection technology.

2 Project Overview
A wide field of view is imperative during controlling a vehicle suchas a car or an airplane or a helicopter. However, the space to set
up windows is limited; consequently, the field of view from thecockpit of the vehicle is insufficient. Therefore, when we drive acar through narrow streets, when we park a car in small parking
lots, and when we back up a car, the operativeness is fairly bad.
Moreover, some accidents are caused due to a narrow field of view
because the drivers cannot recognize a walking person or other obstacles. In addition, a narrow field of view induces a cooped-up
feeling.
Therefore, we propose a transparent cockpit in which the interiorappointments of the vehicle are virtually transparent. In such a
cockpit, the area occluded by the door can be seen as that froman actual window (Figure 1).
∗ E-mail:takumi yoshida@ipc.i.u-tokyo.ac.jp† E-mail:kensei jo@ipc.i.u-tokyo.ac.jp‡ E-mail:kouta minamizawa@ipc.i.u-tokyo.ac.jp§ E-mail:hideaki nii@ipc.i.u-tokyo.ac.jp¶ E-mail:kawakami@star.t.u-tokyo.ac.jp
k E-mail:tachi@star.t.u-tokyo.ac.jp

Figure 2: Implementation of the Transparent Cockpit. The lower
part of a person riding a bicycle can be seen through the interiorappoitments of the car.

The principle of a transparent cockpit is very simple. First, the images captured by cameras (set up outside of the vehicle) are transformed into images as seen from the driver’s viewpoint. Next, thetransformed images are projected from a head-mounted projector(HMP) [Inami et al. 2000] using a retro-reflective projection technology (RPT) [Kawakami et al. 1998]. Then, the objects coveredwith a retro-reflective material are virtually transparent, and the operator of the vehicle can see the outside view through the door, roof,
and so on.

We implemented our proposed system on an actual car. Figire 2shows the view from the driven with wering HMP. The lower partof the person riding a bicycle can be seen through the interior appointments of the car.

Computer

Input Image
Output Image
CameraDriverHMP
Head Position

Figure 3: The Ultimate Goal of this Project.
Retro-reflector

3 Vision
Figure 4: System Diagram of The Transparent Cockpit.

The transparent cockpit can improve the operativeness, safety, andcomfort during controlling the vehicle. For example,• The driver can conveniently drive a car through narrow streets
and back up the car because he or she can see the side of theroad and the back of the car (operativeness).
• The number of accidents would decrease because the driver
can recognize a walking person or other obstacles (safety).
• The driver has a wider visibility of the sky or the landscape;therefore, he or she may experience a liberating feeling as thatobtained while driving a convertible (comfort).
The ultimate goal of the transparent cockpit is that the driver should
not feel as if “I am operating a vehicle,” but “I am running or flying
by myself” (Figure 3). This goal will be achieved in the next few
decades.

4 Technical Innovations
There are three elements necessary for realization of the transparent cockpit; (1) The display that superimpose the outside image ofthe vehicle onto the real world. (2) The way of getting the image
that surrounds the vehicle. (3) To generate appropriate images seenfrom any viewpoint in the vehicle.
We have already developed the “optical camouflage” technology[Inami et al. 2003] corresponding to (1). Now, we propose a system
which enables (2) and (3). By using an imageprocessing technologythat combines the images captured from multiple video cameras andusing the head-mounted projector equipped with a position sensing
device, (2) and (3) are achieved. We describe the detail of the system in following subsections.
In addition, we developed an algorithm for correcting the positionof the virtual screen to correctly project the image of the objects
that are at any distances.

4.1 System Configuration
Figure 4 shows a system diagram of the transparent cockpit. The
process flow of this system is as follows:

1. The multiple wide-angle video cameras are set up outside of
the vehicle capture the images along each direction.
2. To generate a spherical image that surrounds the vehicle, thecaptured images are combined. Then, virtual screen is set up
in a virtual world formulated in a computer and the sphericalimage is projected to the screen.
3. The viewpoint in the virtual world is determined by the operator’s viewpoint measured using the position-sensing device
in the HMP.
4. By applying the correction algorithm, the positions of the virtual screens are corrected.
5. The images seen from operator’s viewpoint that is determinedin 3 are rendered and projected onto the interior appoitments
coverd with the retro-reflective material from the HMP.

4.2

Correction Algorithm using Template Matching

In a conventional optical camouflage system, the distance from theviewpoint to the virtual screen needs to be assumed. Therefore,
when there is a subject at a distance that is different from the assumption, the subject is projected at an incorrect scale and position(Figure 5). In the case of transparent cockpit, the subjects are at various distances (such as a walker, guard rail, buildings, a landscape).
Therefore, the problem is critical in our system.
Then, we propose an algorithm for correcting the position of the virtual screen using template matching. The following is the process
flow of this correcting algorithm (Figure 6). First, a small video
camera is set up on the HMP that operator wear. The camera always faces the same direction as operator. Therefore, the image
captured by the camera is nearly operators view.
“Image A” is captured by the small video camera near the operator’s
viewpoint. The gray-painted area represents the vehicle’s frame
covered with retro-reflective material. The other area is the view
from the window. “Image B” is part of the spherical image generated by the images that the video cameras outside of the vehicle
capture.

㪧㫉㫆㫁㪼㪺㫋㫆㫉㪪㫄㪸㫃㫃㩷㪠㫉㫀㫊㪦㫇㪼㫉㪸㫋㫆㫉㪟㪸㫃㪽㩷㪤㫀㫉㫉㫆㫉㪩㪼㫋㫉㫆㪄㫉㪼㪽㫃㪼㪺㫋㫀㫍㪼㩷㪪㪺㫉㪼㪼㫅

Figure 5: Gap between the Real Object and the Projected Image.
Center of the image is the real object seen from the window of thevehicle. The other is the projected image.

Figure 7: The Principles of RPT.

4.3

Image A

Prototype Implementation

Image BWe did a qualitative evaluation experiment of this system. We actually implemented the prototype of the transparent cockpit on a car.
As a result, it was confirmed that the driver can see the outside viewthat can’t seen in usual cockpit such as the side of the road, traffic
signs on the road, parking lot line, and so on.

Clipping

Template Matching
Template Image

Driver’s View

Figure 6: Correction Algorithm. Image A is captured by a video
camera on the X’tal visor. Image B is part of the spherical image.

5 Context
A head-mounted display (HMD) [Sutherland 1968] or IPT systems
such as CAVE [Cruz-Neira et al. 1993] are famous examples ofvisual displays used in virtual reality systems. They are very useful,but they also have some problems. For example, one of the problem
with HMD is the trade-off between high resolution and a wide fieldof view. On the other hand, IPT system imposes a shadow of theuser’s body on the virtual environment and the interaction of theuser’s virtual body with the user’s real body. Thus, visual displayof virtual reality is still developing. Based on this background, wedeveloped the X’tal vision [Kawakami et al. 1998] at SIGGRAPH
98.

5.1

Retro-reflective Projection Technology (RPT)

X’tal vision employs the RPT that uses an optical projection technique. Figure 7 shows the principles of the RPT. The image of avirtual object is projected through a pinhole. The projected image
is reflected by the retro-reflective screen.
The window area in image A is obtained from the position information of the operator’s viewpoint measured by a position sensing
device on the HMP. First, as a template, an image with a suitablesize is clipped from the window area in image A. Next, it searches
image B for the place where the correlation with the template image is the highest. Then, the correction value is calculated as theposition and scale of the template image in image A correspondswith the template matching result. As a result, the position of thevirtual screen is determined. The position of the virtual screen canbe corrected in real time by performing this process for each frameof the images captured by the video camera.

By using this algorithm, the subject viewed by the operator is projected at the correct distance, and the gap between the real-worldimage and the projected image gets canceled.

Then, at SIGGRAPH 99, we applied a HMP to a visuo-haptic display using a camouflage technique [Inami et al. 2000]. By using
this method, a user can observe a stereoscopic virtual object with anapproximately accurate occlusion relationship between the virtualand the actual environments; the user can actually feel the object.

5.2

Optical Camouflage

Optical camouflage [Inami et al. 2003] is a type of active camouflage that uses RPT (Figure 8). The object that needs to be madetransparent is painted or covered with a retro-reflective material.
Then, a projector projects the background image onto the material
making the masked object virtually transparent. The transparent
cockpit employs the optical camouflage technique.

6 User Experience
We will set up a full-scale cockpit at an exhibition site. The cockpit
has windows, and the outside surroundings can be observed. The
part without the window is covered with a retro-reflective material.
Five or more cameras are set up outside of the cockpit, and theycapture images of the appearance of the exhibition site.
We will modify the HMP; it will be more compact, light, and comfortable. The attendees wear the improved virsion of the HMP;
while sitting in the cockpit, the attendees can experience that thecockpit is actually transparent at the door and ceiling regions. Each
person requires about ten minutes to complete this procedure.
Figure 8: Optical Camouflage [Inami et al. 2003].

References

2QKPV.KIJVC RUZ -N EIRA , C., S ANDIN , D. J., AND D E FANTI , T. A. 1993.
Surround-screen projection-based virtual reality: The design andimplementation of the cave. In Proc. of Computer Graphics,
vol. 27.

+TKU%QPFGPUGT.GPU
4GVTQTGHNGEVKXG5ETGGP.%&2CPGN

2TQLGEVKQP.GPU
1RGTCVQT/KETQ/KTTQTFigure 9: The Principle Construction of the X’tal Visor.

5.3

X’tal Visor

We developed an X’tal visor [Sonoda et al. 2005] at SIGGRAPH
2005. The X’tal visor is a display of the RPT. Figure 9 shows theprinciple construction of the X’tal visor. Here, the large half-mirror
shown in Figure 7 is replaced with an all-reflective micro mirror.
As the mirror size is fairly small, the X’tal visor can display images
without covering the wearer’s face and therefore enables natural
open face-to-face communications.

5.4

Additionally, we will construct a smaller setup to form an easy-touse system. So that many people can experience the transparent
cockpit, each person would take about five minutes to complete theprocedure.

ITS

In recent years, technologies in the form of intelligent transport systems (ITSs) have been developed that support driver’s recognition
capabilities and judgment; these systems have been mounted on
cars. For example, the “night view system” [Toyofuku et al. 2003]
uses near-infrared light to illuminate pedestrians and road topography during nighttime, providing the driver with a wider view thanthat obtained with high beams, regardless of whether the driver isdriving with low or high beams. The transparent cockpit is highly
compatible with these technologies. In the transparent cockpit, information obtained by using these technologies can be displayed on
the doors or the roof.

I NAMI , M., K AWAKAMI , N., S EKIGUCHI , D., YANAGIDA , Y.,
M AEDA , T., AND TACHI , S. 2000. Visuo-haptic display using
head-mounted projector. In Proceedings of IEEE Virtual Reality
2000, 233–240.
I NAMI , M., K AWAKAMI , N., AND TACHI , S. 2003. Optical camouflage using retro-reflective projection technology. In Proceedings of the Second IEEE and ACM International Symposium on
Mixed and Augmented Reality (ISMAR 03).
K AWAKAMI , N., I NAMI , M., M AEDA , T., AND TACHI , S. 1998.
The object oriented displays. In Conference Abstracts and Applications (SIGGRAPH 98).
S ONODA , T., E NDO , T., K AWAKAMI , N., S UZUKI , Y., AND
TACHI , S. 2005. X’tal visor. In Conference Abstracts and Applications of ACM SIGGRAPH2005.
S UTHERLAND , I. 1968. A head-mounted three dimensional display. In Proc. fall Joint Computer Conference, AFIPS Conf.
Proc., vol. 33, 757–764.
T OYOFUKU , K., I WATA , Y., H AGISATO , Y., AND K UMASAKA ,T. 2003. The “night view system” using near-infrared light. InSAE 2003 World Congress & Exhibition.