Pre-print / Post-print version
Best Short Paper Award at ICAT-EGVE2016
International Conference on Artificial Reality and TelexistenceEurographics Symposium on Virtual Environments (2016)
D. Reiners, D. Iwai, and F. Steinicke (Editors)

AquaCAVE: An Underwater Immersive Projection System
for Enhancing the Swimming Experience
S. Yamashita1 , X. Zhang1 , T. Miyaki1 and J. Rekimoto1,2
1 The University of Tokyo, Japan
2 Sony Computer Science Laboratories, Japan

Abstract
AquaCAVE is an underwater immersive projection environment which faithfully reproduces the swimming experience in thevirtual space. AquaCAVE is inspired by the surrounding projection system known as the CAVE Automatic Virtual Environment,
where the stereoscopic images are projected to the surfaces surrounding the user, but addresses several water-specific problems
that were not studied in previous systems. In this paper, we describe techniques to overcome the water-specific issues for configuring the immersive projection system. Three characteristics of water that mainly cause problems are pincushion distortion,reflection, and infrared (IR) radiation absorption. Existing motion capture systems based on IR or blue lights are not feasible
for an underwater immersive projection environment since IR are absorbed, and blue lights disturb a user to see the stereoscopic images. Therefore, we propose a setup for visible-light head tracking, which is functional for AquaCAVE. As a result, theproposed circular polarization-based method was shown to be valid to enable a constantly clear view, stable head tracking, andreflection reduction. With this methodology, we can build the proposed AquaCAVE that can be applicable to future underwater
entertainment and enhanced swimming training.
Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three-Dimensional Graphics andRealism—Virtual reality

1. Introduction
Swimming is one of the best workouts for cardiovascular health.
It is effective for burning calories and puts adequate stress on the
body. On the other hand, to develop good workout habits can bedifficult for beginners. Thus, there have been several attempts to
support the swimming activity with digital technologies such as
Virtual Reality (VR) systems. However, swimming is also a fullbody experience that induces unique perceptual recognitions andfeelings (e.g. water flows, pressure, etc.), which contain complicated integration of various haptic senses over the whole body not
only with visual and auditory sensation. For this reason, reproducing swimming experiences with VR systems is quite challenging
within existing research.
In this study, we introduce an underwater immersive projection
environment. This configuration is inspired by the CAVE Automatic Virtual Environment [CNSD93], where the stereoscopic images are projected to the surfaces surrounding the user. As the images are controlled based on the user’s head position, the environment can provide 3D synthetic experience to the user. In our case,the environment is filled with real water, which faithfully reproduces the swimming experience in a virtual environment. Swimmers can literally dive into the surrounding scenery as if they were
for instance swimming in a real sea with coral reefs (Figure 1). At
c 2016 The Author(s)Eurographics Proceedings c 2016 The Eurographics Association.

Figure 1: AquaCAVE: An underwater immersive projection environment.

the same time, this underwater configuration poses new technicalchallenges which are not addressed by the previous studies.
To configure immersive projection environments, accurate headtracking for detecting eye positions, and projection with appropriate view frustums are required. However, the characteristics of water such as optical distortion and reflection make the necessary con-

S. Yamashita & X. Zhang & T. Miyaki & J. Rekimoto / AquaCAVE: An Underwater Immersive Projection System for Enhancing the Swimming Experience

Reflection	

Pincushion distortion(a) Reflection on the surface of water	 (b) Pincushion distortion in the water	

Figure 2: Examples of issues caused by water.

ditions difficult to satisfy. In addition to these issues, water absorbs
a wide range of infrared radiation (IR). For this reason, IR motion
captures and/or head tracking devices that are widely used in existing VR systems are unusable for our purpose. In this paper, we
describe techniques to overcome the water specific issues for configuring the immersive projection VR system.
2. Related work
Previous research using multimodal feedback enables users to diveinto realistic scenes. Jain et al. [JSG∗ 16] proposed a system that
produces the sensation of diving into the water by using a motion platform with user’s outstretched arms and legs placed in asuspended harness. Haptic turk [CLL∗ 14] is a motion platform toprovide an interactive experience as a flight simulation in VR. To
create the sensation in the virtual world, the system uses tactile
feedback produced by humans. Thus, the representation is limitedby the tools that humans can handle, such as a spray and a fan.
Systems using textiles can not represent sensations that the wholebody senses such as water pressure and flow since the user is notintended to be in a space filled with the water, but air instead. However, AquaCAVE is filled with real water, which faithfully provides
the actual swimming sensation in VR.
Underwater AR displays [MKMK09] and Swimoid [UR13] canshow information in front of the user in the water. Nevertheless,these systems were not able to create the immersive environments
like VR because of the limited size of the displays while the entire
view is covered in AquaCAVE [YZR16].
3. AquaCAVE: Underwater immersive projection system
In this study, we introduce an augmented swimming pool withstereoscopic screens, based on the virtual reality environment withsurrounded rear-projection screens known as CAVE in the water.
In AquaCAVE, a user is able to enjoy the surrounding scenery as
if they were swimming in a real sea with coral reefs. AquaCAVE
can also enhance the swimming experience by showing informationsuch as ideal swimming forms and sensing data in the 3D space.
To implement AquaCAVE, there are several difficulties caused
by water specific issues. Bellow are three characteristics of waterthat mainly cause problems for impressive projection setup.

Infrared radiation (IR) absorption: IR motion capture and headtracking are widely used for VR. However, tracking-devices using IR cannot be used for AquaCAVE, since IR decays greatly
in the water.
Pincushion distortion: Pincushion distortion is an effect from thewater, which causes images to become pinched in the center.
This optical distortion is seen in images taken in the water because of the difference in the refractive index between air andwater.
Reflection: AquaCAVE consists of acrylic panels with rearprojection sheets. The Acrylic panels reflect the images displayed on the other screens. This reflection also occurs on thesurface of water in the pool.
Figure 2 (a) is an example of reflection on the surface of the water,(b) is the pincushion distortion in the water.
4. System implementation
Figure 3 is the system configuration of AquaCAVE. The pool is
3m × 2m × 1m, and made of 3cm thick acrylic panels. The insideof the pool is coated with circular polarizing sheets. Six ultra-shortthrow projectors (RICOH PJ WX4141) are used for the projection.
Each projector is capable of displaying stereoscopic images witha frame sequential method, and all images for projectors are provided from a single Mac Pro with six Thunderbolt display ports.
Stereoscopic images are projected on each wall of the swimming
pool, and goggles with liquid-crystal display (LCD) active shutter
glasses separate images for left and right eyes. In AquaCAVE, theswimmer’s eye position is tracked by the system to correctly projecta 3D scene that can be extended across the boundaries of the poolwalls.
To cope with optical distortion caused by water, image processing like camera calibration [LRL03] is valid. We used a dome-type
lens, which can correct pincushion distortion optically [BBM∗ 11].
4.1. Stable head tracking in the water
Existing motion capture systems based on IR or blue lights do not
fit our purpose. Qualisys Underwater Motion Capture is a motion
capture setup using visible light, which consists of several cameras
emitting blue lights, and passive markers. This system can trackthe 3D positions of the markers in the water. However, since thecameras need to emit strong visible lights to the environment, thestructure is not preferable for the projection environment.
We implemented a visible-light head tracking method, which isfunctional in the water. To achieve both of stable head-tracking
and clear view in AquaCAVE, we used circular polarization as described bellow.
To provide a clear view of the projected images to the user,
we put quarter wave sheets on the active shutter glasses to makethese equivalent to right polarizing sheets. To provide a stable background to the camera, we put a left polarizing sheet on the lens.
Figure 4 shows a light shielding and a transmission by a right circular polarizing sheet on the front screen in the water. Each image
is taken through a left circular sheet (a), and a right circular polarizing sheet (b).
c 2016 The Author(s)Eurographics Proceedings c 2016 The Eurographics Association.

S. Yamashita & X. Zhang & T. Miyaki & J. Rekimoto / AquaCAVE: An Underwater Immersive Projection System for Enhancing the Swimming Experience
Camera 	Rear-projection sheetson acrylic panels	Circular polarizing sheetson the inside of the pool	Eye
Positions	
1m	

Active shutter glasses
on swimming goggles

2m	
Stereoscopic images	
3m	

Projector		

Host computer	Figure 3: The system configuration of AquaCAVE: Stereoscopic images are projected on each wall of the swimming pool. The user wears
3D glasses on swimming goggles in the pool, and the head position is tracked for configuring an immersive environment. The inside of thepool is coated with circular polarizing sheets for a constantly clear view, stable head tracking, and reflection reduction.

Reflection	

Projected screen	(a) An image of front screen taken(b) An image of front screen taken
through a left circular polarizing sheet 	 through a right circular polarizing sheet 	Light shielding	Transmitting	Circular polarizing sheet	(a) An example of reflections onother screens from a projected screen	

(b) Light shielding caused by achange in polarization property 	

Figure 4: Light shielding and transmitting by circular polarization.
Figure 5: Reflection reduction by circular polarization.

The mechanism of the light shielding and transmitting in the poolis described bellow. Right circular polarizing sheets on the screenspolarize the lights from projectors, and the lights are shielded by aleft polarizing sheet on the lens of the camera for head tracking. On
the other hand, the user wearing LCD active shutter glasses can see
the images on the screen clearly, since the glasses are equivalent
to right circular polarization sheets. The reason why LCD active
shutter glasses become right circular equivalent is that the LCD aremade of two linear polarizing sheets with the same axis. If the slow
axis of the quarter wave sheet is at 45◦ to the axis of the polarizer,
the LCD works the same as a right circular polarizing sheet.
4.2. Reflection reduction
Right circular polarization sheets in the pool reduce reflection toother screens significantly. Figure 5 (a) shows an example of thereflection from the front screen. (b) is an image taken through aright circular polarizing sheet. The part of the front screen coatedwith the right polarizing sheet is visible while the reflection is dark.
The reason why this light shielding happens is that the reflected
lights become left-handed circularly-polarized lights. This meansthat the right circular polarization sheet prevents the images frombeing displayed on the other screens. The surface of the water also
reflects images from scenes. This reflection may enhance the swimming experience if the scene is 2D, but causes the same issues for
immersive projection environment. We apply a water current on thesurface to blur the reflection. By this installation, the reflection becomes controllable.
c 2016 The Author(s)Eurographics Proceedings c 2016 The Eurographics Association.

5. Evaluation on light shielding for head tracking
For position detection using visible lights in the water, two maintypes of markers are available. One type of the markers is a fiducial
marker widely used for AR applications. The other is a LED marker
emitting visible lights. We tested the stability and limitation of eachtype of markers.
Natural Feature Tracking (NFT) [NY99] is functional even if
some parts of the marker are covered. This situation is occasionalin AquaCAVE since the user is intended to float in the water using his/her legs and arms. However, the detection of feature points
used in NFT is strongly dependent on the illumination of the scene[MTS∗ 05], [PCZ13]. We put each marker on a human head modeland placed in the projection environment. NyARToolkit for Processing 3.0.2 is used to track an NFT marker. For testing the capacity of the marker, a video taken in the sea filled with coral leafs
is projected on a screen. As a result, the system can track the NFT
marker only in a bright light condition, which unable a clear projection.
LED markers are stable in a dark light condition, but only with asimple background. For testing the capability of LED markers, we
put waterproofed LED markers on a swimming cap, and moved it
in a pool displaying an underwater video. We used a library for processing called BlobDetection for LED marker tracking. The tracking system detects blobs in images with a simple brightness threshold. As a result, the system was not able to detect the LED markers
accurately with any thresholds for luminosity since the backgroundS. Yamashita & X. Zhang & T. Miyaki & J. Rekimoto / AquaCAVE: An Underwater Immersive Projection System for Enhancing the Swimming Experience

7. Conclusion
LED marker	

(a) Front screen with an underwater
scene, affecting blob detection	(b) Circular polarizing sheet on the front
screen, making LED markers recognizable 	Figure 6: Background image stabilization for LED tracking.

image contains a number of blobs, which have a variety of sizes
and brightness as shown in Figure 6 (a). However, the markers became traceable with the light shielding by circular polarization asshown in Figure 6 (b). In other words, the light shielding improved
the stability compared to the situation without polarizing sheets.
6. Discussion and future work
6.1. The effects of water to human perspectives
Objects appear larger in the water than in the air. This expansion and optical distortion may affect the immersive environment.
Therefore, we conducted a test watching a stereoscopic video
through a water tank. As a result, even though the video is slightly
distorted and expanded, observers were able to watch the stereoscopic video as 3D. However, the effects to the stereoscope images
may cause issues to immersive environments, which requires accurate depth information.
In addition, according to a research on human perspectives in thewater [RN03], objects in the water usually appear beyond their optical distance and slightly enlarged in linear size, but in accordancewith size-distance invariance for humans. This discrepancy might
affect the underwater CAVE depending on eye positions and/or
gaze direction. Thus, further investigation on the effects of water
for humans in an immersive projection environment are required.
6.2. Water flow tracking as a user interface
We also conducted a feasibility test of a novel water-flow tracking
method using a photoelastic effect as a user interface for AquaCAVE.
In the test, we used polyethylene (PE) sheets floating on the water in a water tank. We put a polarizing sheet on bottom of thetank, and took a video by using a camera with a polarizing sheet.
Through the camera, we were able to see the colored PE sheets by
the photoelastic effect. As a result, optical flow was valid to track
the movement of PE sheets moving along a water flow in a water tank. This method may enable interactions using water flow in
AquaCAVE. PE sheets are transparent from the view of user and/or
a camera without any polarizing sheets. This means the objects
floating on the surface of water does not affect the view, and 2D
motion capture of the user is still possible in the pool.

We introduced AquaCAVE, an immersive projection environment
in the water. In this study, we investigated water-related issues tomake a CAVE setup, such as IR absorption, pincushion distortion,and reflection. As a result, circular polarization enabled the system
to provide both a constantly clear view and stable head tracking
with a complex background. Moreover, the installation reduces thereflection of the projected images to other screens. However, further investigation on the effects of water for humans is required.
AquaCAVE is applicable to future underwater entertainment andenhanced swimming training.
References[BBM∗ 11] B RUNO F., B IANCO G., M UZZUPAPPA M., BARONE S.,R AZIONALE A.: Experimentation of structured light and stereo vision for underwater 3d reconstruction. ISPRS Journal of Photogrammetry and Remote Sensing 66, 4 (2011), 508–518. doi:10.1016/j.
isprsjprs.2011.02.009. 2[CLL∗ 14] C HENG L.-P., L ÜHNE P., L OPES P., S TERZ C., BAUDISCH
P.: Haptic turk: a motion platform based on people. In Proceedings of the
32nd annual ACM conference on Human factors in computing systems(2014), ACM, pp. 3463–3472. doi:10.1145/2556288.2557101.
2[CNSD93] C RUZ -N EIRA C., S ANDIN D. J., D E FANTI T. A.: Surroundscreen projection-based virtual reality: the design and implementation of
the cave. In Proceedings of the 20th annual conference on Computer
graphics and interactive techniques (1993), ACM, pp. 135–142. doi:
10.1145/166117.166134. 1[JSG∗ 16] JAIN D., S RA M., G UO J., M ARQUES R., W U R., C HIU J.,
S CHMANDT C.: Immersive terrestrial scuba diving using virtual reality. In Proceedings of the 2016 CHI Conference Extended Abstracts
on Human Factors in Computing Systems (2016), CHI EA ’16, ACM,
pp. 1563–1569. doi:10.1145/2851581.2892503. 2[LRL03] L AVEST J.-M., R IVES G., L APRESTÉ J.-T.: Dry camera calibration for underwater applications. Machine Vision and Applications
13, 5-6 (2003), 245–253. doi:10.1007/s00138-002-0112-z. 2[MKMK09] M ORALES R., K EITLER P., M AIER P., K LINKER G.: Anunderwater augmented reality system for commercial diving operations.
In OCEANS 2009 (2009), IEEE, pp. 1–8. 2[MTS∗ 05] M IKOLAJCZYK K., T UYTELAARS T., S CHMID C., Z ISSER MAN A., M ATAS J., S CHAFFALITZKY F., K ADIR T., VAN G OOL
L.: A comparison of affine region detectors. International journal of computer vision 65, 1-2 (2005), 43–72. doi:10.1007/
s11263-005-3848-x. 3[NY99] N EUMANN U., YOU S.: Natural feature tracking for augmented
reality. IEEE Transactions on Multimedia 1, 1 (1999), 53–64. doi:
10.1109/6046.748171. 3[PCZ13] P ŘIBYL B., C HALMERS A., Z EM ČÍK P.: Feature point detection under extreme lighting conditions. In Proceedings of the 28th
Spring Conference on Computer Graphics (2013), ACM, pp. 143–150.
doi:10.1145/2448531.2448550. 3[RN03] ROSS H. E., NAWAZ S.: Why do objects appear enlarged underwater? Arquivos Brasileiros de Oftalmologia 66, 5 (2003), 69–76. doi:
10.1590/S0004-27492003000600009. 4[UR13] U KAI Y., R EKIMOTO J.: Swimoid: a swim support system using
an underwater buddy robot. In Proceedings of the 4th Augmented Human
International Conference (2013), ACM, pp. 170–177. doi:10.1145/
2459236.2459265. 2[YZR16] YAMASHITA S., Z HANG X., R EKIMOTO J.: Aquacave: Augmented swimming environment with immersive surround-screen virtual
reality. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (2016), ACM, pp. 183–184. doi:
10.1145/2984751.2984760. 2

c 2016 The Author(s)Eurographics Proceedings c 2016 The Eurographics Association.