Transparent Cockpit Using Telexistence
Takura Yanagi†,

Charith Lasantha Fernando‡,

Kouta Minamizawa‡,

Susumu Tachi‡,†Mobility Services Laboratory, Nissan Motor Co., JapanMHD Yamen Saraiji‡
Norimasa Kishi§‡Graduate School of Media Design, Keio University, Japan§Senior Innovation Researcher, Nissan Motor Co., JapanABSTRACT
We propose an indirect-vision, video-see-through augmented
reality (AR) cockpit that uses telexistence technology to provide
an AR enriched, virtually transparent view of the surroundings
through monitors instead of windows. Such a virtual view has thepotential to enhance driving performance and experience above
conventional glass as well as head-up display equipped cockpits
by combining AR overlays with images obtained by future image
sensors that are superior to human eyes. As a proof of concept, we
replaced the front windshield of an experimental car by a large
stereoscopic monitor. A robotic stereo camera pair that mimics thedriver’s head motions provides stereoscopic images with seamlessmotion parallax to the monitor. Initial driving tests at moderate
speeds on roads within our research facility confirmed the illusionof transparency. We will conduct human factors evaluations afterimplementing AR functions in order to show whether it is possible
to achieve an overall benefit over conventional cockpits in spite ofpossible conceptual issues like latency, shift of viewpoint andshort distance between driver and display.

Figure 1: Transparent Cockpit concept aims to provide a clear viewof the surroundings even under bad environmental conditions
that can further be augmented to provide visual cues.

Keywords: Augmented Reality, Telexistence, Virtual Reality.

Figure 2: Left: Telexistence. Right:: Augmented transparent cockpitusing telexistence technology.

Index Terms: Computer graphics—Graphics systems andinterfaces—Mixed / augmented reality.
1INTRODUCTION

Drivers obtain up to 90% of information necessary for driving
through the visual channel [1]. Unfortunately, visual information
is often incomplete, for example due to occlusion, darkness orweather conditions. Human processing of the visual channel isalso sequential, so drivers might miss information while theirattention is focused on something else.
Augmented Reality has the potential to enhance driver’sperception and understanding of the surroundings. Past examples
include letting drivers virtually see through occluders, warning
drivers by marking obstacles like pedestrians and providing rivingcues like safe distances or navigational guidance within the main
field of view [2,3]. Such automotive AR systems are mostly either
video see-through using a small display with below-unity
magnification or optical see-through HUDs. Both types of systems
have conceptual limitations: The former requires the driver to lookat yet another display while driving, whereas perfectly registeringand matching AR overlays to appear three dimensional and real isdifficult for the latter.
† takura-y@mail.nissan.co.jp
IEEE Virtual Reality Conference 201523 - 27 March, Arles, France
978-1-4799-1727-3/15/$31.00 ©2015 IEEE

A different problem is solved by indirect vision systems fordriving. Well known examples are night vision enhancement
systems (NVES) for civilian vehicles which show themonochromatic live image of a forward looking infrared camera
on a small LCD display on the dashboard and indirect visionsystems that provide external views to the occupants of a windowless armored vehicle. Human factors evaluations mostly agree inthe potential of NVESs to increase detection distances, but are
divided about their effect on workload. Unity magnification seems
best for distance judgment and speed control, althoughmagnification of less than 1 may help with overall orientation innavigation tasks at the cost of causing slower driving speeds (dueto decreased speed perception) and higher workload [4,5]. In
general, driver task performance increases with driving time,
indicating that there are some limitations that drivers need to getused to. Opinions differ about the presence and amount of
discomfort and contributing factors like motion sickness. Whilemost systems are monocular, a few researchers have investigatedstereoscopic systems (without motion parallax) and concluded thatstereoscopic systems have an advantage in depth perception
resulting in higher task performance, while having no significant
effect in reducing motion sickness [6]. We confirmed in apreliminary driving simulator study that providing motion parallax,
i.e. correctly changing the viewpoint of the displayed image
depending on head position is important to reduce motion sickness.
The aim of our current work is to show the effectiveness of
replicating motion parallax to reduce visual discomfort in indirect
vision cockpits using telexistence technology [7,8,9]. At thismoment, the indirect vision system has been implemented and a311

formal evaluation is planned for the near future. Once that step isdone, we will be able to use the system as a basis to implement atransparent full-size video see-through AR cockpit.
2MOTION PARALLAX INDIRECT VISION SYSTEM

We used a Nissan NV200 minivan as our experimental vehicle.
The front windshield has been removed and a large, stereocapable 60 inch LCD display (Manufacturer: Sharp) was mounted
on top of the firewall and protected by aluminum panels that are
lined in black cloth on the inside to shut out sunlight and eliminate
reflections. (Figure 3)
There are two main approaches to produce seamless motion
parallax video. The option we chose is a multiple DOF robot
camera that exactly tracks the head motion of the driver.
Alternatively, a multiple camera array could be used to interpolate
images for arbitrary viewpoints within the array.
The robotic head system consists of a XY robot (Model: IAI
LSA-S6SS, LSA-S8HS series) and a custom made 3 DOF robotic
head. We decided not to implement Z motion because the driver’s
head does not move much in the vertical direction and a
preliminary driving simulator experiment showed that ignoring Z
motion did not have a significant effect. The head uses two web
cameras fitted with 128.5 degrees wide angle lenses as a stereo
camera pair placed 65mm apart so that it matches the average
human Inter-Pupillary distance (IPD) in order to create correct
distance and depth perception. The completed robot, placed in
front of the backside of the display and above the engine hood,
can be seen in Figure 4. In order to generate control commands forthe robot, the driver’s head motion is captured by a motion
tracking system (Model: OptiTrack Duo).
An initial calibration is used to match the default head positionof the driver to the default position of the robot. After thecalibration, the x, y position as well as pan, tilt and roll of therobotic head is moved so that the stereo camera pair maintains
constant relative position and direction to the drivers head. TheXY robot and the head communicate at 200Hz cycle speed after
filtering rugged motion with a digital low pass filter.
The stereo image pair is mapped onto a virtual projection screen
and the resulting image is shown to the driver wearing an active
shutter 3D glass to which retro-reflective markers for the motion
tracking system were added. Therefore, when the driver moves inx, y direction as well as pan, tilt, roll the robotic head in front of
him moves accordingly to give the exact same Point of Vision.

3

Figure 5: Layout of indirect vision system

REFERENCES
[1]

[2]

[3]

[4]

[5]

[6]

[7]

Figure 3: The windshield of the prototype car was replaced by alarge 3D monitor and enclosed inside a hood to block sunlight.

[8]

[9]

Figure 4: Completed robot camera with the 2 DoF linear servos
above the 3 DoF robot head with the stereo camera.

312

CONCLUSION

We proposed a fully enclosed, windowless, indirect vision,
video see-through AR cockpit as a solution for providing superior
situation awareness by combining advanced image sensors andAR. We believe that motion parallax is essential for illusion oftransparency and implemented a prototype car that replaces thefront windshield by a 5 DoF robotic stereo camera, a headtracking system and a large stereo display. Initial results of driving
it on roads within our research facility have been promising andfurther evaluations are planned for the near future.

A. Cohen, R. Hirsig, “Zur Bedeutung des fovealen Sehens fuer die
Informationsaufnahme bei hoher Beanspruchung,” Sicht undSicherheit im Strassenverkehr, Verlag TUV Rheinland, Koeln, 1990.
U. Bergmeier, C. Lange, “Acceptance of augmented reality for
driver assistance information,” Proceedings 2nd International
Conference on Applied Human Factors and Ergonomics, Las Vegas,
2008.
W. Narzt, G. Pomberger, A. Ferscha, D. Kolb, R. Mueller, J.
Wieghards, H. Hoertner, C. Lindiger, “Augmented reality navigationsystems,” Universal Access in the Information Society 4, 2006, 177–
187.
S. Mahlke, D. Roesler, K. Seifert, J.F. Krems, M. Thuering,“Evaluation of six night vision enhancement systems: Qualitativeand quantitative support for intelligent image processing”, HumanFactors, 49, 518-531, 2007.
C. Smyth, G. Gombash, P. Burcham. “Indirect Vision Driving With
Fixed Flat Panel Displays for Near-Unity, Wide and Extended Fields
of Camera View,”. Tech. rep., Army Research Laboratory, Aberdeen
Proving Ground, MD, June 2001.
J. Chen, R. Oden, C. Kenny, J. Merrit. “Effectiveness ofStereoscopic Displays for Indirect-Vision Driving and Robot
Teleoperation,” Tech. rep., Army Research Laboratory, Aberdeen
Proving Ground, MD, August 2010.
Susumu Tachi, Kouta Minamizawa, Masahiko Furukawa andCharith Lasantha Fernando: “Telexistence - from 1980 to 2012,”
Proceedings of IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS2012), pp.5440-5441, Vilamoura, Algarve,Portugal (2012.10)
Takumi Yoshida, Kensei Jo, Kouta Minamizawa, Hideaki Nii, Naoki
Kawakami and Susumu Tachi, “Transparent Cockpit: Visual
Assistance System for Vehicle using Retro-reflective Projection
Technology,” Proceedings of the IEEE Virtual Reality 2008, pp.
185-188, Reno, USA (2008.3)
Charith Lasantha Fernando, Masahiro Furukawa, Tadatoshi Kurogi,
Sho Kamuro, Katsunari Sato, Kouta Minamizawa and Susumu Tachi,“Design of TELESAR V for Transferring Bodily Consciousness inTelexistence,” Proceedings of IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS2012), pp.5112-5118,
Vilamoura, Algarve, Portugal (2012.10)