Self-Calibrating Depth from Refraction
Zhihu Chen1 , Kwan-Yee K. Wong1 , Yasuyuki Matsushita2 , Xiaolong Zhu1 , and Miaomiao Liu1
1

Department of Computer Science, The University of Hong Kong, Hong Kong
2
Microsoft Research Asia, Beijing, P.R.China
Abstract
cameraIn this paper, we introduce a novel method for depth acquisition based on refraction of light. A scene is captured
twice by a ﬁxed perspective camera, with the ﬁrst image
captured directly by the camera and the second by placinga transparent medium between the scene and the camera. A
depth map of the scene is then recovered from the displacements of scene points in the images. Unlike other existing
depth from refraction methods, our method does not require
the knowledge of the pose and refractive index of the transparent medium, but can recover them directly from the input images. We hence call our method self-calibrating depth
from refraction. Experimental results on both synthetic andreal-world data are presented, which demonstrate the effectiveness of the proposed method.

1. Introduction
Depth from refraction is a depth acquisition method
based on refraction of light. A scene is captured several
times by a ﬁxed perspective camera, with the ﬁrst image
captured directly by the camera and the others by placing atransparent medium between the scene and the camera. The
depths of the scene points are then recovered from their displacements in the images.
Depth from refraction approach has various advantages
over other existing 3D reconstruction approaches. First,
unlike multi-view stereo methods, it does not require calibrating the relative rotations and translations of the camera
as the viewpoint is ﬁxed. Besides, a ﬁxed viewpoint also
makes the correspondence-problem much easier as the projections of a 3D point remain similar across images. Second, unlike depth from defocus methods which require expensive lenses with large apertures to improve depth sensitivity, the accuracy of depth from refraction can be improved by either 1) increasing the thickness of the refractive
medium; 2) using a medium with a large refractive index;or 3) increasing the angle between the viewing direction of
a 3D point and the surface normal of the medium. Third,

scene

(b)
34.2
33.4
32.6
31.8

transparent medium
(a)

(c)

31.0
30.2
[cm]

Figure 1. (a) Experimental setup; (b) a direct image of the scene;(c) reconstructed depth map of the scene.

unlike depth from diffusion methods, which require placing a diffuser with a known orientation close to the object
being measured, depth from refraction allows the transparent medium being placed ﬂexibly between the scene and thecamera.
Existing depth from refraction methods often requireelaborate setup and pre-calibration for accurately knowing
the pose and refractive index of the transparent medium.
These greatly prohibit the applicability of the approach. In
this paper, we introduce a novel method for depth from refraction which is more usable in various scenarios. Our
method requires neither a careful hardware setup nor any
pre-calibration. By simply putting a transparent medium
between a camera and the scene, our method automaticallyestimates the pose and refractive index of the transparent
medium as well as a depth map of the scene.
In our method, a scene is captured twice by a ﬁxed perspective camera, with the ﬁrst image (referred to as the direct image) captured directly by the camera and the second(referred to as the refracted image) by placing a transparent
medium with two parallel planar faces between the scene
and the camera (see Fig. 1). By analyzing the displacements of the scene points in the images, we derive a closed
form solution for recovering both the pose of the transparent medium and the depths of the scene points. Given athird image captured with the transparent medium placed ina different pose, we further derive a closed form solution forrecovering also the refractive index of the medium. We callour proposed method self-calibrating depth from refraction
since it directly exploits information obtained from the images of the scene to determine the necessary parameters for
estimating scene depth.

1.1. Related work
Depth acquisition has a long history in computer vision.
Based on the number of viewpoints required, existing methods can be broadly classiﬁed into multi-view and multiexposure approaches. Multi-view methods exploit stereo
information to recover the depth of a scene [13]. The location of a 3D point can be estimated by ﬁnding and triangulating correspondences across images.
Instead of moving the camera to change the viewpoints,
multi-exposure methods record the scene by changing theimaging process. Depth from defocus methods obtain depth
by exploiting the fact that depth information is contained in
an image taken with a limited ﬁeld of depth: objects at aparticular distance are focused in the image, while objects
at other distances are blurred by different degrees depending on their distances. Pentland estimated a depth map ofa scene by measuring the degree of defocus using one or
two images [12]. In [17], Subbarao and Gurumoorthy proposed a simpler and more general method to recover depth
by measuring the degree of blur of an edge. Surya and Subbarao [18] used simple local operations on two images taken
by cameras with different aperture diameters for determining depth. Zhou et al. [20] pointed out that the accuracy ofdepth is restricted by the use of a circular aperture. They
proposed a comprehensive framework to obtain an optimized pair of apertures. All the aforementioned methods require large apertures to improve depth sensitivity. Recently,
Zhou et al. [19] proposed a depth from diffusion method.
Their method requires placing a diffuser with known orientation near the scene. They showed that while depth fromdiffusion is similar in principle to depth from defocus, it can
improve the accuracy of depth obtained with a small lens byincreasing the diffusion angle of a diffuser.
Our work is more closely related to [6, 9, 3, 4, 14, 15].
In [6], Lee and Kweon obtained the geometry of an object using a transparent biprism. In [9] and [3], the authors estimated the depth of a scene using a transparent planar plate with 2 opposite faces being parallel to the image
plane. In [4], Gao and Ahuja considered the case where thefaces of the transparent medium are not parallel to the image plane, and images are captured by rotating the medium
about the principal axis of the camera. They estimatedthe pose and the refractive index of the medium in an extra step using a calibration pattern. In [14, 15], Shimizu
and Okutomi proposed reﬂection stereo which records the

Figure 2. When a light ray passes from one medium to anotherwith different speeds, it bends according to Snell’s law. This phenomenon is known as refraction of light.

scene from a ﬁxed viewpoint with and without the reﬂection
light paths. The two light paths create individual images,
and from these images, their method estimates depth by triangulation. Their setup uses either reﬂective or refractive
medium for the implementation. However, their method requires a complex calibration setup as described in [16]. Our
method as well uses a transparent plane-parallel medium;
however, unlike the aforementioned methods, our method
requires neither the medium plane being parallel to the image plane, nor a careful calibration of the pose and refractive
index of the medium using a calibration pattern.
Shape recovery of transparent objects (also referred toas refractive objects) has also attracted the attentions ofmany researchers [11, 1, 10] recently. Instead of targetingat reconstructing refractive surfaces, this paper, on the other
hand, exploits a refractive object to obtain the depth of ascene.
The rest of the paper is organized as follows. Section 2
brieﬂy reviews the theory on refraction of light. Section 3
describes our proposed self-calibrating depth from refraction method in detail. Experimental results on both synthetic and real data are presented in Section 4, followed byconclusions in Section 5.

2. Refraction of Light
Refraction of light refers to the change in the directionof a light ray due to a change in its speed. This is commonly observed when a light ray passes from one medium
to another (e.g., from air to water). The refractive index ofa medium is deﬁned as the ratio of the velocity of light invacuum to the velocity of light in the said medium. Consider a light ray L1 originated from a point P passing froma medium M1 to another medium M2 (see Fig. 2). Let therefractive indices of M1 and M2 be n1 and n2 respectively,
and the interface between the two media be a plane denotedby Π1 . Suppose L1 intersects Π1 at S1 with an angle ofincidence θ1 . After entering M2 , L1 changes it directionand results in a refracted ray L2 with an angle of refraction
θ2 . By Snell’s law, the incident ray L1 , the surface normal
at S1 and the refracted ray L2 are coplanar, and the angle ofincidence θ1 and the angle of refraction θ2 are related by
n1 sin θ1 = n2 sin θ2 .

(1)

After travelling for some distance in M2 , L2 leaves M2 andenters M1 again. Let the interface between M2 and M1 be
a plane denoted by Π2 which is parallel to Π1 . Suppose
L2 intersects Π2 at S2 and after entering M1 , it changes its
direction and results in a refracted ray L3 . Since Π1 andΠ2 are parallel, it is easy to see that the angle of incidence
for L2 is θ2 . It follows from Snell’s law that the angle ofrefraction for L3 is θ1 , and L1 , L2 , L3 and the surface normals of Π1 and Π2 are coplanar, with L1 parallel to L3 .
Hence, the refraction plane of P (formed by L1 , L2 andL3 ) is perpendicular to both Π1 and Π2 .
It is worth noting that total internal reﬂection might happen when a light ray passes from a medium with a higher
refractive index to one with a lower refractive index. The
critical angle is deﬁned as the angle of incidence that results in an angle of refraction being 90◦ . When the angle
of incidence is greater than the critical angle, total internal
reﬂection occurs and the light ray does not cross the interface between the two media but is reﬂected totally back in
the original medium.

3. Depth from Refraction
In this section, we will derive a closed form solution forrecovering the depth of a scene from the displacements ofscene points observed in two images due to refraction oflight. As mentioned in Section 1, a scene will be captured
twice by a ﬁxed perspective camera, with the ﬁrst image(referred to as the direct image) captured directly by thecamera and the second (referred to as the refracted image)by placing a transparent medium between the scene and thecamera. We assume the intrinsic parameters of the camera
are known, and the transparent medium consists of two parallel planar faces through which light rays originate fromscene points enter and leave the medium before reaching
the camera.

3.1. Medium Surface  Image Plane
Consider a 3D point P being observed by a camera centered at O (see Fig. 3). Let the projection of P on the image
plane be a point I. Suppose now a transparent medium M
with two parallel planar faces is placed between P and thecamera in such a way that the two parallel planar faces areparallel to the image plane. Due to refraction of light, P
will no longer project to I. Let I be the new image position for the projection of P. By considering the orthogonal

Figure 3. Special case of shape from refraction in which the parallel planar faces of the medium are parallel to the image plane. The
depth of a 3D point can be recovered in closed form.

projection of the line OP on the image plane, we haved tan α = (d − w − u) tan θ1 + w tan θ2 + u tan θ1 , (2)where d is the depth of P, α is the angle between the visual ray of P and the principal axis of the camera (also referred to as the viewing angle of P), w is the thickness ofthe medium M , u is the shortest distance between O andM , and θ1 and θ2 are the angle of incidence/refraction andangle of refraction/incidence as the ray originated from P
enters/leaves M . Rearranging Eq. (2) gives
d=w

tan θ1 − tan θ2.
tan θ1 − tan α(3)By Snell’s law, we havesin θ1 = n sin θ2 ,(4)

where n is the refractive index of M and the refractive index
of the air can be approximated to one. Let r be the distance
between I and I . It can be expressed in terms of the focal
length f of the camera and the angles θ1 and α asr = f (tan θ1 − tan α).

(5)From Eq. (4) and Eq. (5), we derive expressions for tan θ1
and tan θ2 asr+ tan α,(6)tan θ1 =ftan2 θ1.
(7)tan θ2 =2n + (n2 − 1) tan2 θ1Finally, substituting Eq. (6) and Eq. (7) into Eq. (3) gives




f
1
d = w 1 + tan α.
1−rn2 + (n2 − 1)( fr + tan α)2(8)Figure 4. The formation process of the refracted image cannot be
described by a simple pinhole camera model.

From Eq. (8), it can be seen that d does not depend on u.
This is very important in practice as it implies that the depthof a scene point can be recovered without knowing the distance between the medium M and the camera.
The proposed depth from refraction method has a majordifference with traditional multi-view approaches as the formation process of the refracted image cannot be describedby a simple pinhole camera model. In Fig. 4, L is a light
ray originated from a point P to the camera center O, and
L is perpendicular to the medium surface. Light ray L1
originated from P1 intersects L at point O1 , while light ray
L2 originated from point P2 intersects L at point O2 . The
distance between O and O1 is given by
1 − sin2 θ1tan θ2. (9)ds1 = w − w=w 1− tan θ1n2 − sin2 θ1
From Eq. (9), it can be seen that ds1 depends on θ1 . Therefore O1 and O2 are two different points. It follows thatthe formation process of the refracted image cannot be described by a simple pinhole camera model.
It has been pointed out in Section 2 that total internal
reﬂection can only happen when a light ray passes from one
medium with a higher refractive index to one with a lower
index. In our setup, light rays travel through a composite
air-glass-air medium. Total internal reﬂection should notoccur when a light ray enters the glass from air as its speed
in air is faster than its speed in glass. Since total internal
reﬂection does not occur when the light ray enters the glass,the incident angle must be less than the critical angle when
it re-enters air from the glass. Hence total internal reﬂection
should never occur in our setup.

3.2. Medium Surface ∦ Image Plane
It has been shown in the previous subsection that depth
can be recovered using Eq. (8) when the parallel planar
faces of the transparent medium are parallel to the image
plane. However, this proposed setup has two major limitations. First, it is difﬁcult to ensure that the parallel planar

Figure 5. When the parallel planar faces of the medium are not parallel to the image plane, the depth of a 3D point can be recoveredin closed form if the orientation of the medium is known.

faces of the medium are parallel to the image plane. Second, the depths of those 3D points whose viewing angles
are small will be very sensitive to noises (see Fig. 7 (a)). As
a result, only points with large viewing angles (i.e., thoseprojected near the border of the image) can be accurately
recovered. In this subsection, we will show how the closed
form solution derived under the special case can be applied
to recover depth in the general case where the parallel planar faces of the medium are not parallel to the image plane
but have a known orientation.
Suppose the surface normal of the parallel planar faces ofthe medium is given by the unit vector N, and the viewing
direction of the camera is given by the unit vector V (seeFig. 5). A rotation about the optical center of the camera,
with a rotation axis given by the cross product of V and
N and a rotation angle given by the angle between V andN, will bring the image plane parallel to the parallel planar
faces of the medium. Such a rotation will induce a planar
homography that can transform the image of the original
camera to an image observed by the camera after rotation.
The closed form solution derived in the previous subsection
can then be applied to the transformed image to recover thedepth dv of a point P with respect to the rotated camera
using the viewing angle αv of P and the displacement rvof the projections of P in the rotated camera. Finally, thedepth d of P in the original camera can be obtained byd=dv
cos α.
cos αv(10)

3.3. Recovering Pose of the Medium
It has been shown in the previous subsection that scene
depth can be recovered given the orientation of the parallel
planar faces of the transparent medium. In this subsection,
we will show how the orientation of the parallel planar facesof the medium can be recovered directly from the displacements of the scene points in the images.

some correspondences, more correspondences can then be
established with ease using the refraction line constraint derived from J.

3.4. Estimation of the Refractive Index

Figure 6. Recovering the orientation of the parallel planar faces of
the transparent medium. A refraction line deﬁned by corresponding points in the direct and refracted images contains the vanishingpoint for the normal direction of the parallel planar faces of themedium.

Consider a 3D point Q whose visual ray is perpendicular to the parallel planar faces of the medium (seeFig. 6). By construction, the visual ray of Q will simply
pass straight through the medium without any change of direction. Hence, the projections of Q will be identical inboth the direct image and refracted image. Let us denote
this point by J. Without loss of generality, consider another
3D point P, and let I and I be the projections of P in the
direct image and the refracted image respectively. Referring to Section 2, the refraction plane of P is perpendicular
to the parallel planar faces of the medium. Since this plane
contains both P and O, and that the ray OQ is, by construction, also perpendicular to the parallel planar faces ofthe medium, it follows that Q must also lie on this plane.
This plane intersects the image plane along a line which werefer to as a refraction line. It is obvious that both J, I andI must lie on this line. Based on this observation, we havethe following proposition:
Proposition 1. Given the correspondences of two or morescene points in a direct image and a refracted image, the refraction lines deﬁned by the correspondences will intersect
at a single point which corresponds to the vanishing point
for the normal direction of the parallel planar faces of themedium.
The two corollaries below then follow directly from
Proposition 1:
Corollary 1. The orientation of the parallel planar faces of
the transparent medium can be recovered as the visual ray
for the point of intersection between two or more refractionlines deﬁned by correspondences in the direct image and therefracted image.
Corollary 2. Given the projection I of a scene point P
in the direct image, its correspondence I in the refracted
image must lie on the half-inﬁnite line I (t) = I + t(I − J)
where t ≥ 0 and J is the the vanishing point for the normal
direction of the parallel planar faces of the medium.
Having recovered the vanishing point J for the normal
direction of the parallel planar faces of the medium fromIn the previous discussions, we have assumed that therefractive index of the transparent medium is known. In thissubsection, we will show the refractive index of the medium
can be recovered from the displacements of scene points inthree images.
Consider a 3D point P. Let I be its projection in a direct image, and I and I be its projections in two refracted
images captured with the transparent medium positioned in
two different poses respectively. Let d be the depth of P
estimated from I and I using Eq. (10), and d be the depthof P estimated from I and I using Eq. (10). Now by equating d with d , the refractive index n of the medium can berecovered. In practice, given m pairs of correspondencesin three images, the refractive index of the medium can be
estimated by
n = argminnm(di − di )2 .

(11)

i=1

Note that a similar minimization can also be used to estimate the refractive index when there are more than two
refracted images captured with the transparent medium positioned in different poses.

4. Experiment
The methods described in the previous sections for recovering the pose and refractive index of the medium, andthe depth of the scene have been implemented. Experiments
on both synthetic and real data were carried out and the results are presented in the following subsections.

4.1. Synthetic Experiment
Noise analysis was carried out in the case when the parallel planar faces of the medium were parallel to the image plane. Uniformly distributed random noise was added
to the pixel coordinates with noise levels ranging from 0
to 3 pixels. For each noise level, 1000 independent trials
were performed. The experimental setup consisted of some
3D points being viewed by a synthetic camera with a focal length of 70mm. In the ﬁrst experiment, the thicknessof the medium was 4cm and its refractive index was 1.4.
Experiments were carried out for distinct viewing angles.
Fig. 7 (a) shows a plot of the root mean square (RMS) of
the relative depth errors against different noise levels fordifferent viewing angles. It is shown that for a particular
noise level, depth accuracy could be improved by increasing the viewing angle in the case when medium surfaces

3.0

3.5
25°

40°

55°

1.4

1.7

2.0

2.3

2cm

2.0
1.5
1.0
0.5

relative depth error [%]

3.0

2.5

0.0
0.0

2.5
1.1

70°

relative depth error [%]relative depth error [%]

10°

2.5
2.0
1.5
1.0
0.5

0.5

2.0
1.0
1.5
noise level [pixel]

2.5

3.0

0.0
0.0

0.5

2.0
1.0
1.5
noise level [pixel]

2.5

3.0

4cm

6cm

8cm

10cm

2.0
1.5
1.0
0.5
0.0
0.0

0.5

1.0
1.5
2.0
noise level [pixel]

2.5

3.0

(a)
(c)
(b)
Figure 7. Depth accuracy can be improved in three ways: (a) increasing the angle between the visual ray and the surface normal of themedium; (b) increasing the refractive index; or (c) increasing the thickness of the medium.

are parallel to the image plane. In the general case, it implies that depth accuracy could be improved by increasing
the angle between the visual ray and the surface normal ofthe medium. In the second experiment, the viewing angle
and thickness of the medium were 25◦ and 4cm, respectively. Experiments were performed for different refractive
indices. Fig. 7 (b) shows a plot of the RMS of the relative
depth errors against different noise levels for different refractive indices. It is shown that for a particular noise level,
depth accuracy could be improved by increasing the refractive index. For a refractive index of 1.4, the relative depth
error is less than 1.0% under a 3-pixel noise level. In thethird experiment, the viewing angle and refractive index of
the medium were 25◦ and 1.4, respectively. Experiments
were performed for different thicknesses of the medium. It
is shown in Fig. 7 (c) that for a particular noise level, depth
accuracy could be improved by increasing the thickness ofthe medium.
In the synthetic experiment of estimating the orientationof the parallel planar faces of the medium, a bunny model
was captured using a synthetic camera and a 4cm thicktransparent medium with a refractive index of 1.4. Noise
with levels ranging from 0 to 3 pixels were added to the
pixel coordinates of 8171 points. For each noise level, 100
independent trials were performed. The orientation of theparallel planar faces of the medium and the 3D coordinatesof all the points were obtained using correspondences in the
direct image and the refracted image. Fig. 8 (a) shows the
RMS angular error of the estimated normal of the parallel
planar faces, as well as the RMS of the relative depth errors
against different noise levels. The reconstruction result witha noise level of 1.0 pixel is shown in Fig. 8 (c, d).
In the synthetic experiment of estimating the refractive
index of the medium, a third image of the bunny model was
captured with the transparent medium positioned in a different pose. The orientations of the parallel planar faces ofthe medium were estimated using correspondences between
the direct image and each of the two refracted images. The
refractive index of the medium and the 3D coordinates of

all the points were then obtained using correspondences inall three images. Fig. 9 (a) shows the RMS error of the estimated refractive index and the RMS of the relative depth
errors against different noise levels. The reconstruction result with a noise level of 1.0 pixel is shown in Fig. 9 (c,d).

4.2. Real Experiment
A common transparent glass (i.e., it is not of opticalgrade) with parallel planar faces and a thickness of 4.9cm(measured by a common ruler) was used in all the real experiments. In fact, it follows from Eq. (8) that if the thickness is unknown, the shape could also be obtained up to ascale. In the ﬁrst real experiment, one direct image and two
refracted images of a model building were captured by acamera. The intrinsic parameters of the camera were calibrated using [2]. Correspondences of the corners on the surface of the model were picked manually. The direct imageof the model with corners marked by red crosses is shownin Fig. 10 (a). More correspondences among the three images were obtained by SIFT feature matching [8]. SIFT
was ﬁrst performed to obtain matches between the direct
image and the ﬁrst refracted image. RANSAC [5] was thenused to eliminate outliers. In using RANSAC, two pairs ofmatches were sampled to recover the orientation of the parallel planar faces of the medium as the intersection of thetwo refraction lines deﬁned by the correspondences, and inliers were identiﬁed by computing the distance of the intersection point from the refraction lines deﬁned by each pairof correspondences. The same procedure was performed toobtain correspondences between the direct image and thesecond refracted image. The matches among three images
were then obtained by combining correspondences between
images. The poses and the refractive index of the medium,and the 3D coordinates of the points were estimated using
all the correspondences (obtained manually and by SIFT)
among three images. The model building generated using
the reconstructed corners is shown in Fig. 10 (b, c).
The experimental results were also quantitatively compared against those obtained using a calibration pattern in0.7
relative depth error
angular error

1.2

0.6

1.0

0.5

0.8

0.4

0.6

0.3

0.4

0.2

0.2

0.1

0.0
0.0

0.0
3.0

0.5

1.0
1.5
2.0
noise level [pixel]

2.5

angular error [degree]

relative depth error [%]

1.4

(b)

(a)

(c)

(d)

Figure 8. Reconstruction of the bunny using a direct image and a refracted image. (a) Estimation of the orientation of the parallel planar
faces of the medium and the depths of points on the surface of the bunny model; (b) the original bunny model; (c, d) two views of thereconstructed bunny model.

7

0.08
relative depth error
error of refractive index

0.07

6

0.06

5

0.05

4

0.04

3

0.03

2

0.02

1

0.01

0
0.0

0.5

1.0
1.5
2.0
noise level [pixel]

(a)

2.5

error of refractive index

relative depth error [%]

8

0.00
3.0

(b)

(c)

(d)

Figure 9. Reconstruction of the bunny using a direct image and two refracted images captured with the transparent medium in two different
poses. (a) Estimation of the refractive index of the medium and the depth of points on the surface of the bunny model; (b) the original
bunny model; (c, d) two views of the reconstructed bunny model.

[3], which were treated as the ground truth. In the experiment, depth of each corner in the pattern could be computed by taking a direct image of the calibration pattern.
The ground truth of the ﬁrst orientation of the medium andthe refractive index could then be estimated by using thedepths of the corners in the pattern. After calibration, themodel building was placed in the view of the camera without moving the camera and medium. The angle between
the estimated normal of the medium and the ground truthis 4.05◦ . The ground truth refractive index is 1.438, andthe estimated index is 1.435. The reconstructed 3D points
were re-projected to the three images. The RMS of the reprojection errors is 0.726 pixel.
In the second real experiment, one direct image and one
refracted image were used to obtain the depth map of thescene. The orientation of the medium surface was estimated from the correspondences in two images obtainedusing SIFT. SIFT ﬂow [7] was then used to obtain dense
correspondences across the two images. From the dense
correspondences, our method can recover a high-precision
result of a cat model, as shown in Fig. 11. Fig. 12 shows
the result of a more complicated scene recovered using thesame procedure. Note that the artifacts in both ﬁgures werecaused by the inaccuracy of correspondences obtained by
SIFT ﬂow method.

(a)

(b)

(c)

Figure 10. The reconstruction result of a model building. (a) A
direct image of the model with corners marked by red crosses;(b,c) two views of the reconstructed model.
37.9
37.0
36.0
35.1
34.1

(a)

(b)

(c)

(d)

33.2
[cm]

Figure 11. The reconstruction result of a model cat. (a) A direct
image; (b, c) two views of the reconstructed cat model; (d) depth
map of the model.

71.7
68.6
65.5
62.4
59.3

(a)

(b)

(c)

(d)

56.2
[cm]

Figure 12. The reconstruction result of a scene consisting of two
toy models. (a) A direct image; (b) depth map of the scene; (c, d)
two views of the reconstructed scene.

5. Conclusions
A self-calibrating depth from refraction method is introduced in this paper. It is demonstrated that a transparent
medium with parallel planar faces can be used to recoverscene depth. Two images of a scene are captured by a camera with and without placing a transparent medium between
the scene and the camera. Correspondences in images arethen used to obtain the orientation of the parallel planar
faces of the medium and the depths of scene points. It isfurther pointed out that a third image with the medium positioned in another orientation could be used to estimate therefractive index of the medium. Experiments on both synthetic and real data show promising results. With the proposed method, the pose and refractive index of the transparent medium, and depths of scene points can be estimatedsimultaneously. Nevertheless, the proposed method suffersfrom the same intrinsic limitation as other existing depth
from refraction methods: it corresponds to a small baseline
multi-view approach. Hence the proposed method could
only be used to recover depth for close scenes.

References[1] S. Agarwal, S. P. Mallick, D. Kriegman, and S. Belongie. On
refractive optical ﬂow. In Proc. European Conf. on Computer
Vision, pages 483–494, 2004. 2[2] J.-Y. Bouguet. Camera calibration toolbox for matlab.
http://www.vision.caltech.edu/bouguetj/
calib_doc/. 6[3] C. Gao and N. Ahuja. Single camera stereo using planar
parallel plate. In Proc. Int. Conf. on Pattern Recognition,
volume 4, pages 108–111, 2004. 2, 7

[4] C. Gao and N. Ahuja. A refractive camera for acquiring
stereo and super-resolution images. In Proc. Conf. Computer
Vision and Pattern Recognition, volume 2, pages 2316–2323,
2006. 2[5] R. I. Hartley and A. Zisserman. Multiple View Geometryin Computer Vision. Cambridge University Press, second
edition, 2004. 6[6] D. Lee and I. Kweon. A novel stereo camera system by abiprism. IEEE Transactions on Robotics and Automation,
16:528–541, 2000. 2[7] C. Liu, J. Yuen, and A. Torralba. Sift ﬂow: Dense correspondence across scenes and its applications. IEEE Trans.
on Pattern Analysis and Machine Intelligence, 2010. 7[8] D. Lowe. Object recognition from local scale-invariant features. In Proc. Int. Conf. on Computer Vision, volume 2,
pages 1150–1157, 1999. 6[9] H.-G. Maas. New developments in multimedia photogrammetry. Optical 3-D Measurement Techniques III, 1995. 2[10] N. Morris and K. Kutulakos. Dynamic refraction stereo. In
Proc. Int. Conf. on Computer Vision, volume 2, pages 1573–
1580, 2005. 2[11] H. Murase. Surface shape reconstruction of a nonrigid transport object using refraction and motion. IEEE Trans. on
Pattern Analysis and Machine Intelligence, 14:1045–1052,
1992. 2[12] A. P. Pentland. A new sense for depth of ﬁeld. IEEE Trans.
on Pattern Analysis and Machine Intelligence, 9:523–531,
1987. 2[13] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, andR. Szeliski. A comparison and evaluation of multi-view
stereo reconstruction algorithms. In Proc. Conf. Computer
Vision and Pattern Recognition, volume 1, pages 519–528,
2006. 2[14] M. Shimizu and M. Okutomi. Reﬂection stereo - novel
monocular stereo using a transparent plate. In Proc. on
Canadian Conference on Computer and Robot Vision, pages
CD–ROM, 2006. 2[15] M. Shimizu and M. Okutomi. Monocular range estimation
through a double-sided half-mirror plate. In Proc. on Canadian Conference on Computer and Robot Vision, pages 347–
354, 2007. 2[16] M. Shimizu and M. Okutomi. Calibration and rectiﬁcation
for reﬂection stereo. In Proc. Conf. Computer Vision and
Pattern Recognition, pages 1–8, 2008. 2[17] M. Subbarao and N. Gurumoorthy. Depth recovery fromblurred edges. In Proc. Conf. Computer Vision and Pattern
Recognition, pages 498–503, 1988. 2[18] G. Surya and M. Subbarao. Depth from defocus by changing camera aperture: a spatial domain approach. In Proc.
Conf. Computer Vision and Pattern Recognition, pages 61–
67, 1993. 2[19] C. Zhou, O. Cossairt, and S. Nayar. Depth from diffusion. In
Proc. Conf. Computer Vision and Pattern Recognition, 2010.
2[20] C. Zhou, S. Lin, and S. Nayar. Coded aperture pairs for depth
from defocus. In Proc. Int. Conf. on Computer Vision, pages
325–332, 2009. 2