[Poster] Turbidity-based Aerial Perspective Rendering for Mixed Reality
Carlos Morales*

Takeshi Oishi*

Katsushi Ikeuchi*

The University of TokyoABSTRACT
In outdoor Mixed Reality (MR), objects distant from the observer
suffer from an effect called aerial perspective that fades the colorof the objects and blends it to the environmental light color. The
aerial perspective can be modeled using a physics-based
approach; however, handling the changing and unpredictableenvironmental illumination is demanding. We present a turbiditybased method for rendering a virtual object with aerial perspective
effect in a MR application. The proposed method first estimates
the turbidity by matching luminance distributions of sky models
and a captured omnidirectional sky image. Then the obtained
turbidity is used to render the virtual object with aerial perspective.
Keywords: Photorealistic rendering; MR/AR for art, cultural
heritage, or education and training.
Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, augmented, andvirtual realities; I.3.7 [Computer graphics]: Three-dimensional
Graphics and Realism—Color, shading, shadowing, and texture
1

INTRODUCTION

Outdoor MR consists of integrating virtual object into real scenes
in a way that the object’s appearance corresponds to the realscene’s look. When the target object viewed by the observer is far,the perceived object’s appearance loses contrast and becomesblurred. This natural effect is known as aerial perspective and isdue to the light scattered by particles in the atmosphere. Therefore,in such MR applications, we need to render an artificial aerial
perspective over the virtual object to emulate the natural
atmospheric effect. However, handling with the changing andunpredictable natural atmospheric phenomena such as theenvironmental illumination and weather conditions, while
implementing a fast rendering algorithm, is still challenging.
Representative works related to aerial perspective modelling
and rendering are available in the literature. McCartney [1]presented an excellent review that contains relevant data about thescattering phenomena under different weather conditions
categorized by the heuristic parameter turbidity (T). T relates thescattering by molecules of air and particles of haze and provides aclassification of atmospheric conditions, such as T=2 for a clearatmosphere, T=7 for a hazy atmosphere, or T>20 for foggy
conditions. Preetham et al. [2] proposed a turbidity-based
analytical sky model for various atmospheric conditions and used
it to render aerial perspective in Virtual Reality (VR) applications.
Dobayashi et al. [3] proposed a fast rendering approach togenerate atmospheric scattering effects using graphics hardware incomplete-virtual applications. Nielsen [4] presented a real time
*e-mails: {carlos, oishi, ki}@cvl.iis.u-tokyo.ac.jp

rendering system for simulating atmospheric scattering effects forVR. Narasimhan and Nayar [5] proposed a physics-based
scattering model to describe the appearances of real scenes underuniform bad weather conditions. Riley et al. [6] presented alighting model for rendering various optical phenomena in virtualapplications. Schafhitzel et al. [7] rendered planets withatmospheric scattering effects in real time. Bruneton and Neyret[8] presented a real-time method for rendering both sky and aerial
perspective from all viewpoints (ground to outer space) in VR.
In order to achieve photometric consistency in MR, modelingthe scattering phenomena and achieving real-time rendering arerequired. In this work, turbidity is estimated from omnidirectionalsky images and used in an improved full-spectrum aerial
perspective model to render compelling appearances for MR.
2ATMOSPHERIC TURBIDITY ESTIMATION

Turbidity is estimated by matching the luminance distribution of
turbidity-based Preetham sky models and an omnidirectional sky
image captured by a fish-eye lens. First, the sun position at thecaptured sky image is estimated by either finding the center of thesaturated area of the sun or using the longitude, latitude, date andtime at the observer’s position. Then we calculate the luminance(Y from the XYZ color space) ratio Yi/Yref between a sampling
point and a reference point in the captured image, and we computethe ratio Yi(T)/Yref(T) at the corresponding points in Preetham sky
models with the same sun position. The turbidity-based sky modelthat best matches the captured sky image is the one where thedifference between both ratios, that is the ratio error Err, is zero.
Hence, we estimate the turbidity by minimizing the following
equation at N sampling points using the Levenberg-Marquardt
algorithm (LMA):
NYi (T )Y− i .
Y(T)
Yi=1
ref
refarg min ∑T ∈1,20[ ](1)Since the Preetham sky model does not provide equations forcalculating the brightness of cloudy pixels, the Random Sample
Consensus (RANSAC) approach is used to remove cloudy pixels(outliers) from the sampling and estimate turbidity only fromclear-sky pixels (inliers).
3

AERIAL PERSPECTIVE RENDERINGThe intensity value Ic for channel c∈{r,g,b} recorded by a camera
provides the colour based-rendering equation that integrates thespectral light going through the camera lens and the camera’s
spectral sensitivity qc over the visible spectrum (380 to 780nm). In
outdoor scenes, the total light perceived by the camera (observer)
can be modelled as a summation of a direct transmission and anairlight. We approximated the spectral sensitivity by a Dirac’sdelta function to obtain the intensity value Ic of a virtual object’s
pixel perceived by an observer at distance s asI c (s) = I 0,c ⋅ Γc (T , s) + I ∞,c (T ,θ )(1 − Γc (T , s)) ,(2)where I0,c is the intensity value of a pixel at the surface of thevirtual object, and Γc is the approximate attenuation factor:

780nmΓ c (T, s) =∫ e

− β sc ( λ ,T,h0 )sqc (λ )d λ380nm

images captured by Canon EOS5D with a fish eye lens. Figure 1
shows our results with rendering speed of 1800 pixels/sec.
.

(3)

780nm

∫ q (λ )d λc380nm

We benefit from qc estimated in [9]. I∞,c is the intensity value ofa pixel at an infinite distance in the viewing direction with zenith
angle θ . We approximated I∞,c at any viewing direction using [2]:⎡1 + (0.1787 T − 1.4630 )e(−0.3554 T + 0.4275 ) / cos θ a ⎤I ∞a , c = I ∞b , c ⎢⎥,⎢⎣1 + (0.1787 T − 1.4630 )e(−0.3554 T + 0.4275 ) / cos θ b ⎥⎦(4)

where the upper indexes “a” and “b” stand for the pixel at thetarget object and the pixel at the visible sky portion respectively
that share the same azimuth.
4PROPOSED IMPROVED SCATTERING MODEL FOR MR

The total scattering coefficient βsc is a summation of the Rayleighand Mie scattering coefficients, βR and βM respectively. We
propose an improved scattering model that ensures a compelling
result when implementing the aerial perspective for MR. For thispurpose, we employed real measurements of [1] and adapted aturbidity-based classification of the scattering coefficients byconsidering T=1.6 as the turbidity for an exceptionally clear
atmospheric condition, and linearly fitting βR and βM to [1].
Our modified Rayleigh scattering coefficient is given by− h08π 3 (n 2 − 1) 2 ⎛ 6 + 3 p n ⎞H0 ,⎜⎟βR =⎜ 6 − 7 p ⎟K R × e3Nλ4n
⎝
⎠

(5)

where λ is wavelength, n, pn, H0, and N are same as in [10] and h0
is the height at the observer. We calculated a straightforward
multiplicative correction factor KR =1.0396 based on [1].
Our modified Mie scattering coefficient is given byv −2− h0⎛ 2π ⎞(6)β M = 0.434c(T )π ⎜ ⎟ K M × e H 0 ,⎝ λ ⎠where υ and H0 are same as in [11]. Using [1], we propose aconcentration factor c(T)=(0.65T-0.65)×10-16 that guarantees anon-scattering case for T=1 and a corrected Fudge factor
KM=0.0092 adapted to MR.
5EXPERIMENTAL RESULTS

All the experiments were implemented in C++ with a PC (OS:Windows 7; CPU: Corei7 2.93GHz; RAM: 16GB). We used theVirtual Asukakyo project [12] that employs MR technology torestore the ancient capital of Japan. We rendered the CG model ofAsukadera, an ancient temple in Asukakyo, on a fixed view in thecurrent Tokyo city. The height at the observer position was
h0=40m. The distance from the rendered Asukadera to theobserver was about 3500m. Turbidity was estimated from sky

6CONCLUSION

We proposed an efficient turbidity-based method for rendering
virtual objects with aerial perspective effect in MR. Our method
estimates turbidity by matching the luminance distributions of asky model and an omnidirectional captured sky image. We havealso proposed an improved turbidity-based scattering model forMR. Our model benefits from [1] to classify scattering coefficientvalues via turbidity. We use this enhanced scattering model toprovide a full-spectrum aerial perspective rendering model. Ourrendering approach benefits from [6] to go from radiance light to
RGB colors. Although our rendered results applied to VA did notreach real time, this can be easily accelerated in the future via
GPU implementation. Besides, the fixed view issue can behandled using conventional tracking systems.
ACKNOWLEDGEMENTS
This work is partially supported by NET Project of MEXT, Japan.
REFERENCES[1]E. McCartney. Optics of the atmosphere: scattering by moleculesand particles. John Wiley and Sons, 1975.
[2] A. Preetham, P. Shirley, and B. Smits. A practical analytic model fordaylight. In ACM Transactions on Graphics (SIGGRAPH), 1999.
[3] Y. Dobayashi, T. Yamamoto, and T. Nishita. Interactive rendering ofatmospheric scattering effects using graphics hardware. Graphics
Hardware, pp. 1-10, 2002.
[4] R. Nielsen. Real time rendering of atmospheric scattering effects forflight simulators. Master’s thesis, Informatics and Mathematical
Modelling, Technical University of Denmark, DTU, 2003.
[5] S. Narasimhan and S. Nayar. Contrast restoration of weather
degraded images. In IEEE Transactions on Pattern Analysis andMachine Intelligence, Vol. 25, No. 6, June 2003.
[6] K. Riley, D. Ebert, M. Kraus, J. Tessendorf, and C. Hansen. Efficient
rendering of atmospheric phenomena. Eurographics Symposium on
Rendering, 2004.
[7] T. Schafhitzel, M. Falk, and T. Ertl. Real-time rendering of planetswith atmospheres. WSCG International Conference in Central Europe
on Computer Graphics, Visualization and Computer Vision, 2007.
[8] E. Bruneton and F. Neyret. Precomputed atmospheric scattering.
Eurographics Symposium on Rendering, Vol. 27, Number 4, 2008.
[9] R. Kawakami, H. Zhao, R. Tan, and K. Ikeuchi. Camera spectral
sensitivity and white balance estimation from sky images.
International Journal of Computer Vision, Vol. 105, Issue 3, pp.
187-204, 2013.
[10] L. Rayleigh. On the scattering of light by small particles. In
Philosophical Magazine 41, pages 447-454, 1871.
[11] G. Mie. Beiträge zur Optik trüber Medien, speziell kolloidaler
Metalllösungen. Annalen der Physik 4, 377–445, 1908.
[12] The
VirtualAsukakyoProject:
http://www.cvl.iis.utokyo.ac.jp/research/virtual-asukakyo/

Figure 1. Rendering results. From left to right. Virtual Asukadera model without aerial perspective effect. Results of Virtual Asukadera renderedwith our aerial perspective effect for T=2.10. Rendering result for T=2.94. Rendering result for T=4.36.