Shape from Second-bounce of Light Transport
Siying Liu1 , Tian-Tsong Ng1 , and Yasuyuki Matsushita2
1

Institute for Infocomm Research Singapore
2
Microsoft Research Asia

Abstract. This paper describes a method to recover scene geometry
from the second-bounce of light transport. We show that form factors(up to a scaling ambiguity) can be derived from the second-bounce component of light transport in a Lambertian case. The form factors carryinformation of the geometric relationship between every pair of scene
points, i.e., distance between scene points and relative surface orientations. Modelling the scene as polygonal, we develop a method to recover
the scene geometry up to a scaling ambiguity from the form factors by
optimization. Unlike other shape-from-intensity methods, our method simultaneously estimates depth and surface normal; therefore, our method
can handle discontinuous surfaces as it can avoid surface normal integration. Various simulation and real-world experiments demonstrate thecorrectness of the proposed theory of shape recovery from light transport.

1

Introduction

Interreflections, reciprocal reflections among reflecting surfaces, are observed in
all real-world scenes. The way light transports varies with scene geometry andsurface reflectance. Clearly, there is a mutual dependency between the lighttransport and scene environment. This fact is used for scene modeling, e.g.,
by Nayar [1] for scene geometry and reflectance, and also by Yu et al . [2] andMachida et al . [3] for modeling bidirectional reflectance distribution functions(BRDFs), when prior knowledge of the scene is available (pseudo geometry andreflectance for [1], and accurate scene geometry for [2, 3]). Recent advances incomputational photography enabled modeling of inverse light transport [4–7]from photographs without prior knowledge of the environment. These works
open up a new open problem — can we infer scene geometry only from the lighttransport without any prior knowledge?
In this paper, we propose a new approach to inferring scene geometry fromthe measured light transport without using any prior knowledge about the scene.
We focus our discussion on a Lambertian case and model the scene as composedof planar patches. Our approach can be viewed as an inverse radiosity method
where the scene geometry is unknown a priori as illustrated in Fig. 1. We firstshow a form factor matrix, which represents how much light is transported fromone scene point to another purely by geometric factor, up to a scaling ambiguity,
can be obtained from the second-bounce of light transport. Using the form factor

2

S. Liu, T. -T. Ng and Y. Matsushita
Forward: Rendering

𝜃𝑖𝑟𝑖𝑗

Geometry𝜃𝑗Form Factor

Light Transport

Backward: Geometry recovery (our method)

Fig. 1. The relationship between scene geometry and light transport. The forward
case is a rendering process where light transport can be computed from known scene
geometry, while the backward case is where the scene geometry is inferred from lighttransport.

matrix, we show that scene geometry can be recovered up to a scaling ambiguity.
We develop a solution method for simultaneously estimating surface orientations
and scene depths from the form factor matrix by optimization.
The primary contributions of this paper are twofold. First, it introduces theuse of the second-bounce of light transport for recovering scene geometry. We
describe the relationship between the scene geometry and the light transportand show what information is carried in the second-bounce component about
the scene. To this end, we show that the scene geometry can be recovered up
to a scaling ambiguity as well as diffuse albedo ratios. Second, the proposed
method is effective even when the surface of interest has discontinuity. Unlike
prior shape-from-intensity methods, our method simultaneously estimates surface orientation and depth (up to scaling ambiguity). This allows us to avoid
integration of surface orientations; therefore, the assumption of continuous surface is no longer needed unlike other shape-from-intensity methods.
1.1

Prior work

Forward light transport is well studied in computer graphics such as in ray tracing [8] and radiosity [9, 8]. These use known scene geometry and BRDFs for
producing photorealistic images. More recently, photographic modeling of forward light transport is drawing attention [10–13]. These methods take a numberof images under different lightings for recording various complex lighting effects.
In graphics, inverse global illumination was introduced by Yu et al . [14] for
estimating reflectance properties, rather than for geometry estimation. Inverse
light transport is also used in computer vision. Seitz et al . [4] showed a method
for estimating n-bounce component of light transport by probing a scene using
a narrow beam light. Ng et al . [7] extended the method using a stratified matrix
inversion for radiometric compensation of projector-camera systems. Nayar et
al . [5] proposed a fast method for separating direct and global component ofShape from Second-bounce of Light Transport

3

light transport using high frequency illumination. Gupta et al . [6] later discussed
the relation between global illumination and defocused illumination.
The goal of this paper is shape recovery from the measured light transport.
The closest to our work is Nayar et al . [1]. They proposed an iterative photometric stereo algorithm, which is the first work that uses interreflections as a
useful cue for shape and reflectance estimation. Our method is different fromtheir approach in that we infer scene geometry directly from the second-bouncecomponent of light transport instead of relying on photometric stereo. In addition, our method is not limited to continuous surfaces because our method does
not require integration of the surface orientations, but simultaneously estimatessurface orientation and depth. On the other hand, compared with their method,
our method requires more images as input for obtaining the second-bounce component of the light transport.
Apart from shape-from-intensity methods, other prior art on shape or depth
recovery include shape from structured light [15] and structure-from-motion(SfM) [16]. Both approaches use triangulation for determining depths. In termsof calibration requirements, these methods require calibration of intrinsic parameters of the imaging devices, while our method does not require intrinsic
calibration.
Our method uses form factors for shape estimation. The computation of form
factors has a long history back to Lambert in 1760 [17]. Schröder and Hanrahan
derived a closed-form solution for the case of general polygons [18]. Our method
uses form factors in an inverse manner for estimating the scene geometry.

2

Interreflection and Scene Geometry

2.1

Forward case: the Rendering Equation

The rendering equation [19] is written asZcos θi cos θodAp0 ,
Lout (p, ωo ) = Le (p, ωo )+ρ(p, ωi , ωo )Lout (p0 , −ωi )V(p, p0 )kp − p0 k2
M2(1)
where Lout (p, ωo ) is the reflected or outgoing radiance in direction ωo , Le is theemission corresponding to light sources, ρ is the Bidirectional Reflectance Distribution Function (BRDF) of the scene, and V is the binary visibility function.
The visibility function V (p, p0 ) is 1 if scene points p and p0 are connected bya line of sight and 0 otherwise. The integral is over the area of M 2 of all scene
surfaces, and weighted by a purely geometric factor known as the form factor.
The above rendering equation applies for a continuous surface. Discretizationof the surface leads to a matrix representation. For a surface with n facets,33

In this paper, we use the term “facet” to describe the smallest piece of a surface
subdivision and the term “patch” for any larger pieces, up to and including thebiggest polygons formed by combining facets.

4

S. Liu, T. -T. Ng and Y. Matsushita

radiance and albedo values are assumed to be constant over each facet, then therendering equation can be written in operator notation as [20]:lout = le + KGlout = le + Alout , where A = KG.

(2)lout is a vector of Lout (p, ωo ), le is a vector of Le (p, ωo ), G is a purely geometric
operator that takes outgoing or reflected radiance and propagates it within thescene to obtain incident radiance, and K is a local linear reflection operator
based on the BRDF of the surface:






ρ1 0 · · · 0
0 G12 · · · G1n
0 ρ1 G12 · · · ρ1 G1n 0 ρ2 · · · 0  G21 0 · · · · · ·  ρ2 G21 0 · · · · · · 






K= . . .
,G =  .
,
A
=
 ..


.
..
.
..  .
..
.. . . . ..  ..
 .
 .. .. . . 0 .
.
. 
0 0 · · · ρn

Gn1 · · · · · ·ρn Gn1 · · ·

···

0
(3)
The interreflections between two points or facets pi and pj can be described by
the Gij expression4 :
Gij =

0

b i )(bbj )V (pi , pj )(−brij · nrij · nV (pi , pj ) cos α cos β=,krij k2
krij k2(4)

where rij = pj − pi , α and β are the angles between rij and their respective
surface normals. Gii is undefined for any i, and Gij vanishes if pi and pj aremutually invisible.
2.2

Backward case: From Light Transport T to Form Factor G

Following [19] and Eq. (2), we can obtainlout = (I − A)−1 le .

(5)

Assuming the camera does not see the light source directly, and we do not haveemissive surfaces, we can replace le with the effective emission that corresponds
to the direct reflection from the light source, le = Flin , where lin is the incident
light from a light source such as a projector, and F is the light transport matrix
that corresponds to the first-bounce reflection. Assuming a focused light source,
the first-bounce matrix F is diagonal. Hence, we havelout = (I − A)−1 Flin = Tlin , T = (I − A)−1 F,(6)where T is the light transport matrix. Hence, we can write A in terms of T as
A = I − FT−1 .
4(7)

In [1], the geometric kernel takes into account the effective area of the illuminatorfacet. Here, we assume facets i and j are interchangeably the illuminator and reflector
and have sufficiently small area s.t. Gij and Gji are approximately equal.

Shape from Second-bounce of Light Transport

5With Neumann series expansion, we can expand a light transport matrix into amatrix series where the second term corresponds to the second-bounce:
T = (I − A)−1 F = (I + A + A2 + · · · )F.

(8)

We can see that the matrix A is related to the second-bounce of light transport.
According to [4], for a Lambertian scene, the diagonal elements of F are givenby the reciprocals of the diagonal elements of T:
F[i, i] =1.
T−1 [i, i](9)Therefore, if we limit our discussion to the Lambertian case, A can be computed
using Eq. (9) and Eq. (7). For geometry estimation, we can extract G from A.
Given A, we can compute the relative albedo ρij for all the scene points 5 :ρi Gij
ρi
. Aij
=
= .
ρij =
Aji
ρj Gjiρj(10)

Given ρij , we can recover G up to a scale:
A = K̃G̃ and G̃ = K̃−1 A,

(11)

where



ρ1j 0 · · · 0 0 ρ2j · · · 0 1K̃ = K =  . . .
,ρj .. .. . . 0 
0 0 · · · ρnj

3

and G̃ = ρj G.

(12)Geometry extraction from the geometric form factors

For two mutually visible points 6 p1 and p3 as shown in Fig. 2 (a), the geometric·bn1 )(br13 ·bn3 )b1, nb 3 ) = (−br13kr
form factor is given by G13 (r13 , n. For mutually visibility,2
13 kb 1 ) > 0 and (bb 3 ) < 0. Resolving nb i and rij for all
we need to have (br13 · nr13 · nscene points recovers both depth and surface normal. However, this method is
unable to recover geometry for scene points which are not visible by others, e.g.,geometry extraction is impossible for a globally convex surface. In this section,
we will examine the settings under which the scene geometry can be extracted.
5

As an observation, given the relative albedo ρij , we can recover the absolute albedo
value for all scene points as long as the absolute albedo value of one of the scene
points is known.
6
A discrete surface is composed of small facets that are often assumed to have uniform
property. Hence, a discrete facet is conceptually similar to a discrete point. For easeof discussion, we may use the term “facet” and “point” interchangeably.

6

S. Liu, T. -T. Ng and Y. Matsushita
n3

n1

p3

p1

p

1

Π

12

r 21

p

2

3

1

Π12

n3
r 31

p

r 31

p

(a) A two-point case

n1

n3

n1

p

p2

4

Π 34
r 63

r 16

p

3

p

4

r 34

Π34

p5

(b) A two-patch case

p6

Π56

(c) A three-patch caseFig. 2. Three different setup for geometry extraction from the geometric form factor
terms.

3.1

Problem Setup and Assumptions

In this work, we consider a light transport acquisition setup with a projectorcamera system. Assuming no serious scattering due to the transmission media or
subsurface scattering, the directional nature of the projector light allows correspondence between the projector pixels and scene points to be established. The
directional light from the projector is perspective in nature. However, if we assume that the scene depth is small, the projection is approximately orthographic.
With this assumption, the problem of geometry extraction is greatly simplified,
as we can assume that the correspondence points in the scene approximately
preserve the grid structure of the projector pixels. In a coordinate frame where
the z-axis is aligned with the optical axis of the projector, we can assume thatthe x-y coordinate of the scene points form a rectangular grid, which is known
up to a scale, while the z-coordinate is the only unknown.
3.2

A case with two scene points

In the case with just two scene points as in Fig. 2 (a), knowing the value of theform factor is not sufficient to recover the surface normal nor the depth uniquely.
To see the set of all possible solutions, we can rewrite Eq. (4) asb
b1 = −
r13 · nG13 kr13 k2.
bb3
r13 · n(13)

In our setting, for the scene point position in R3 , only the z-coordinate is unknown. As Eq. (13) can only be unique up to a relative depth in z-direction,
there is no loss of generality to fix the z-coordinate for one of the scene point,
say p3 . Then, the distance vector is governed by z1 , i.e., the z-coordinate of p1 ,b 1 in Eq. (13), it is possible to find a r13 for all nb 3 . As nb 1 andalone. For every nb 3 are unit normal vectors, they live in a spherical space S2 . Hence, the space ofnb 3 ) ∈ S2 × S2 , which is highly ambiguous. This also indicatesall solutions (bn1 , nthat having more independent pairs of points does not help, as the normals aretotally unconstrained.

Shape from Second-bounce of Light Transport

3.3

7

A case with two patches

To resolve the ambiguity in the form factor expression, we need to introduce moreconstraints. One way to do so is to group a set of adjacent points to form a patch
where the points share a common surface normal. Fig. 2 (b) shows an exampleof two patches, each having two points. The newly introduced constraints areb1 = nb2,
nb3 = nb4,
nbb 1 = 0, and bb 3 = 0.
r12 · nr34 · n(14)

With the four scene points, we have four distinctive and non-zero form factor
b1,
terms, i.e., G13 , G14 , G23 and G24 . Altogether, there are 5 unknowns, i.e., nb3, bnr12 , br13 and b
r34 , with 7 degrees of freedom, where b
rij has only 1 degree of
freedom as we fix the (x, y) coordinate. Given the 6 equations, the solution spaceis 1 dimensional. The system is sufficiently constrained if we add another point
to either patch, as it introduces 1 additional unknown but 3 more equations.
Therefore, in our algorithm, we group 3 points in a patch.
3.4

A case with three patches

As shown in Eq. (12), we can only obtain the form factor up to an unknown
albedo value, therefore the actual expression for the form factor term for two
mutually visible points p1 and p3 isb1, nb3) = C
G13 (r13 , nb 1 )(bb3)(−b
r13 · nr13 · n,2
kr13 k(15)

where C is an unknown constant. In the case of uncalibrated projector, theconstant C is also needed to account for the unknown scale inherent in the (x, y)
coordinate for scene points.
To disambiguate C, we check for the geometric consistency with three patches
as shown in Fig. 2 (c). As the solution obtained by evaluating the form factor terms for patch pairs (Π12 , Π34 ) and (Π12 , Π56 ) should agree with that for(Π12 , Π56 ), we can validate the solution of the former with that of the latter.
The solutions should best tally when we choose a correct constant C. In thesetting of Fig. 2 (c), there are 9 unknowns with 12 degrees of freedom while
having 15 equations gives a sufficiently constrained solution space. Without any
assumption, the constant C in Eq. (15) is fundamentally unresolvable, as there
is a physically feasible surface geometry with a different albedo corresponding
to a C. However, with the orthographic assumption mentioned in Sec. 3.1, theconstant C is no longer linearly related to depth. With an incorrect C, geometry
can be inconsistent in the triangular patch configuration of Fig. 2 (c). Hence,
the orthographic assumption breaks the scale ambiguity.

4

Algorithm

As shown in Sec. 3, we need to group at least three scene points into patches inorder to obtain a sufficiently constrained system. As another issue, the geometry

8

S. Liu, T. -T. Ng and Y. Matsushita

derived from disjoint point sets will be in different coordinates. In this section, we
will look into the criterion for point grouping and the way to bring the geometryat disjoint coordinates into the global coordinate. We will also look at efficient
ways for geometry extraction through hierarchical computation or incorporating
the prior knowledge of planar surface in the scene.
4.1

Point grouping

The assumption and guiding principle for point grouping is essentially based on
the co-planar property of a point set. An arbitrary set of points is not guaranteed
to be co-planar. Hence, we select points that satisfy the following criterion:– Adjacency: The points are adjacent to each other.
– Mutual invisibility: Two points with Vij = 0 are not mutually visible. Vij = 0
implies Gij = 0.
Two points satisfying the above criterion are likely to be co-planar. In our implementation, we consider points in a 2 × 2 neighborhood as being adjacent.
4.2

Pairwise patch selection

To make the form factor expression in Eq. (4) more succinct, for three co-planarb 1 , we can express
points p1 , p2 , and p3 sharing a common unit normal vector nb 1 asn
r12 × r23b1 =, where rij = pj − pi .
(16)nkr12 × r23 k
If there is an additional point p4 on the same patch, we need to introduce a
constraint to ensure co-planarity:(r12 × r23 ) · r24 = 0.

(17)

As two points on a patch are having Gij = 0 to begin with, we assume that theconstraints such as Eq. (17) are automatically satisfied and will not form part ofthe equations that we are solving. Hence, for two mutually visible patches with
N points each, there are N × N equations for Gij with 2N unknowns whichcorrespond to the z-coordinate of the 2N points, thus forming a sufficiently
constrained system.
In practice, as Gij ’s are obtained from measurement, the Gij ’s with a low
intensity tend to have a low signal to noise ratio and should not be used forcomputation. As a result, we can have fewer equations while the number of
unknowns remains unchanged. To ensure that a patch pair forms a constrained
system, we use the following criteria to select a pair of patches with Na and Nb
points respectively:
Nb
Na X
X
i=1 j=1

1(Gij > ) ≥ Na + Nb where 1(true) = 1 and 1(false) = 0(18)

Shape from Second-bounce of Light Transport

9

Algorithm 1 “Closed-loop” check for reconstructed patches
Require: A list L of valid patch pairs based on Eq. (18)
1: Sort L in descending order of the sum of entries in G for all patch pairs.
2: Start with any patch from the first pair in L and treat it as a parent patch.
3: Record visited patches into a list Lv . List is blank for the initial patch.
4: repeat5:
For a parent patch, identify all its possible branches.
6:
Find the best branch based on the sorted list L. If the best branch is in Lv , takethe next one.
7:
Reconstruct the patch pair and update the depth map and normal map.
8:
Push the traversed patch into Lv .
9:
Use the traversed patch as parent and Goto 1.
10: until The branch patch is the same as the starting patch.
11: Compute the depth error between the initial and final patches.

In our algorithm, we reconstruct the geometry for a pair of patches at a time
and then bring the resultant disjoint geometry into the same coordinate frame
through the common points connecting the different pairs of patches. However, if
there is no direct or indirect visibility link between two points, bringing them into
a common coordinate is impossible. The condition for the existence of a global
coordinate for all points is that the form factor matrix G forms a fully connected
graph. For every point in a common coordinate frame, we verify its geometry byexamining the depth consistency in the closed paths associated to the point. Suchclosed paths could be many, therefore we only consider the one with the highest
intensity and involving at least two other points on different patches. In thisprocess, we are able to identify the reliability of the geometry reconstruction fora point. This consistency check through a “closed-loop” algorithm is presented
in Algorithm 1. It is also intended to disambiguate the unknown constant as
described in Sec. 3.4.
To increase the reliability of geometry estimation, we perform the abovementioned steps in a hierarchical manner, from a finer resolution to a coarser
one. At one level, we estimate the geometry and group points with similar normals into patches. The patch size grows with increasing level, hence the systemof equations for pairwise reconstruction gets more and more constrained andproduces more reliable estimation.

Fast method for piece-wise planar scenes: If the scene is known to bepiece-wise planar a priori, it is more efficient to adopt a top-down approachfor the reconstruction. Except for convex surfaces that do not interact with eachother, planar surfaces correspond to “blocks” of zeros. It is worth-noting that theform factor matrix resembles the weight matrix W in the Normalized Cut [21]problem. With this observation, we can segment the scene into planar surface.

10

S. Liu, T. -T. Ng and Y. Matsushita

Fig. 3. From left to right: A simulated “M” scene with 11 × 23 facets and its G matrix;
a simulated inverted “V” scene with discontinuity made of 12 × 28 facets and its G
matrix. The inner wedge of the “M” scene is made up of 2 convex planes which do notilluminate each other. Note the discontinuity between the 2 planes in the inverted “V”
scene. The form factor matrices are log-scaled for display purposes.

Fig. 4. Top row: Reconstruction results for both clean and noisy simulated “M” andinverted “V” scenes. Bottom row: The recovered surface normal corresponding to thescenes in the top row (normal plotted in opposite directions for display purpose).

5

Experimental Results

To verify our theoretical results, we performed experiments on both synthetic
and real data. For synthetic scenes, the reconstruction is based on simulated
form factor matrices; while for the real data, the light transport T of the scene
is measured and the form factor matrix G is derived from T.
5.1

Synthetic scene

For this experiment, we focus on recovering the shape of simulated 3-D models.
To demonstrate the robustness of the proposed method, we perturbed the form
factor matrix by additive Gaussian noise. Fig. 3 shows the simulated models andtheir corresponding form factor matrices.
Fig. 4 shows the reconstruction results for both simulated scenes, using bothclean and noisy data. In the noiseless case, perfect recovery of both surface normal and depth can be achieved. Observe that the scale of the reconstructionis the same as the data as we begin with a form factor matrix in the scene’sShape from Second-bounce of Light Transport

11

Table 1. Shape recovery result. Normal RMSE and angular error between a pair ofsurfaces are shown. (all errors are measured in degree)
Scene
Normal Angular RMSE ∠ planes 1&2 ∠ planes 2&3 ∠ planes 3&4“M” (clean)
0.00180.0
0.0
0.0“M” (noisy)
14.13
5.60
4.30
10.20
inv. “V” (clean)
0.02
0.0
inv. “V” (noisy)
13.11
6.57
-

coordinate frame. The recovered structures are subject to a translation in the zdirection as the global depth reference point was arbitrarily set. For the noisy
case, both form factor matrices are corrupted by zero-mean Gaussian noise ofstandard deviation 0.5. In the presence of noise, the shape is better recovered atplaces such as the joint of 2 planes where the interreflections are stronger. As
compared to the recovered depth values, the surface normals are better recovered because they are common among all facet pairs in a particular system of
equations. For performance evaluation, we computed the angular error between
all estimated surface normals and their ground truth. We also compared the recovered angles between planes with their ground truth in the simulated models.
The results are presented in Table. 1.
Handling surface discontinuity: To highlight the proposed method’s strengthin handling depth discontinuity, we simulated an inverted “V” scene with a gap
in the center. Fig. 4 shows the successful reconstruction of this scene. Unlike
most shape-from-intensity methods which require the surface to be continuous
for the integration of surface orientation, our method is not restricted by surface
continuity. Facets lying on occluding boundaries do not have interactions withthe rest and therefore do not form any valid equations with them. As a result,these facets will be left unreconstructed since there is insufficient information todetermine their relative positions from the others. The same applies to facets
lying on the joint between 2 planes. As it is co-planar with both planes, its form
factor with facets on both planes equates to 0.
Handling constant factor in G: The constant factor C in Eq. (15) can be
determined empirically through closed-loop checks. As we have fixed the x-, ycomponents of the distance vector and evaluate only the z-component, such a
scaling would cause a non-linear change in z. If this factor is not compensatedfor, the error will show up in the closed-loop check as it propagates through all
pair-wise depth estimation before looping back to the starting patch. The error
here is defined as the minimum depth error among all close-loop paths. Hence,
we can conduct a coarse-to-fine 1-D search to determine the correct factor tocancel C off. To see how C affects the reconstruction, we simulated the recovery
of a “V” scene by fitting in different values. The results are presented in Fig. 5.
Note that C is being multiplied into the form factors in this experiment but in12

S. Liu, T. -T. Ng and Y. Matsushita
Closed loop error E vs. C

C = 0.6

C=1

C = 1.4

0.25
0.2

E

0.15
0.1
0.05
0
0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

C

Fig. 5. Top left: A plot of closed-loop error versus C. Second from top left onwards: Reconstruction results by setting C to various values. The recovered shape gets distorted
as C deviates from 1.

reality we seek to find a reciprocal to compensate for C. As C deviates from 1,
the distortion causes planar surfaces to bend as z changes non-linearly with C.

5.2

Real-world scene

In this experiment we seek to recover the shape of a real-world scene from its
measured light transport matrix. For experimental setup, we used a Canon 5D
camera and a Dell 2400MP projector. In our experiment, we consider grayscale
light transport for simplicity by assuming that the projector-camera color mixing
matrix is diagonal. To ensure interreflections is faithfully measured, we used HighDynamic Range capturing with 12 stops of exposures to acquire T by a bruteforce method. The acquired T is verified by bounce separation. In general, thelight transport matrix T obtained by a projector-camera system has a dimensionof Nc × Np , where Nc and Np are respectively the number of camera pixels andprojector pixels. In this work, we establish a pixel mapping between the camera
and the projector by corresponding a camera pixel to the projector pixel thatinduces a maximum response on it. In our setup, there are more than one camera
pixels being mapped to a projector pixel and we group these camera pixels toform a super-pixel. The intensity of a super-pixel is given by the mean of thegroup of camera pixels. With super-pixels, the resulting T matrix takes a squaredimension of Np × Np . If a super-pixel corresponds to a facet in the scene, withthe pixel grouping procedure, we are inherently making uniform-intensity facet
assumption.
Fig. 6(a) shows the result of a real “M” scene. The angle between planes 1
and 2 is 80o and that between planes 3 and 4 is 55o . (b) shows the derived form
factor matrix G. For the real data, we first determine the unknown scale factor
C (= 0.5) and multiply G by C1 . Fig. 6(c) and (d) show the recovered normal
map and shape; (e) shows the final result after plane fitting. The reconstructedshape is quite close to the original scene. The estimated angle between planes 1
and 2 is 70.43o and that between planes 3 and 4 is 50.55o , giving rise to angularerrors of 9.57o and 4.45o respectively.

Shape from Second-bounce of Light Transport

(b)

(a)

(d)

13

(c)

(e)

Fig. 6. (a) An image of an “M” scene. (b) The derived form factor matrix G. (c) and(d) The recovered surface normal (plotted in opposite direction for display purpose)and shape. Closed loop error is minimized when C = 0.5. (e) The recovered shape after
plane fitting.

6

Conclusion and Discussions

In this paper, we present a method to estimate the scene geometry, i.e., both
the depth and the surface normal simultaneously, from a light transport matrix obtained with a projector-camera system. This method can handle a scene
with discontinuity. We focused on extracting the geometry information fromthe second-bounce component that encodes scene interreflections. This method
works on convex surface with strong interreflections, which often makes the conventional shape-from-intensity methods fail. Ideally, a complete algorithm forgeometry estimation from a light transport matrix should also make use of thefirst-bounce component, which will help on convex portion of a scene and complement our method. We leave the complete algorithm to future work. Lighttransport is often applied for relighting applications that assume static lighttransport. The capability to estimate geometry will open up opportunities infast acquisition of dynamic-scene light transport and make light transport editing possible for graphics applications. In future, we will look into more robust
signal processing techniques to improve the shape reconstruction.
Limitations. One limitation of the proposed reconstruction algorithm lies in
the concavity of the scene. Standalone convex surface cannot be reconstructed.
However, if there exist other surfaces in the scene forming concave pairs with it,
the geometry of this locally convex surface can still be recovered, e.g., the innerwedge of the “M” scene can be reconstructed despite its convex nature, as the 2
inner planes interact with the outer planes to form concave pairs.

References
1. Nayar, S., Ikeuchi, K., Kanade, T.: Shape from interreflections. Proc. of Int’l Conf.
on Computer Vision (1990) 2 – 11

14

S. Liu, T. -T. Ng and Y. Matsushita

2. Yu, Y., Debevec, P., Malik, J., Hawkins, T.: Inverse global illumination: recoveringreflectance models of real scenes from photographs. In: Proc. of ACM SIGGRAPH.
(1999) 215–224
3. Machida, T., Yokoya, N., Takemura, H.: Surface reflectance modeling of real objects
with interreflections. In: Proc. of Int’l Conf. on Computer Vision. (2003) 170–177
4. Seitz, S.M., Matsushita, Y., Kutulakos, K.N.: A theory of inverse light transport.
Proc. of Int’l Conf. on Computer Vision (2005) 1440–1447
5. S. K. Nayar, G. Krishnan, M.D.G., Raskar, R.: Fast separation of direct andglobal components of a scene using high frequency illumination. Proc. of ACM
SIGGRAPH (2006) 935–944
6. Gupta, M., Tian, Y., Narasimhan, S.G., Zhang, L.: (de) focusing on global lighttransport for active scene recovery. Proc. of Computer Vision and Pattern Recognition (2009)
7. Ng, T.T., Pahwa, R.S., Tan, K.H., Bai, J., Quek, T.Q.S.: Radiometric compensation using stratified inverses. Proc. of Int’l Conf. on Computer Vision (2009)
8. J. D. Foley, A. van Dam, S.K.F., Hughes, J.F. In: Computer Graphics: Principles
and Practice. (1990)
9. Goral, C. M., T.K.E.G.D.P., Battaile, B.: Modelling the interaction of light between diffuse surfaces. Computre Graphics 18 (1984) 212–222
10. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proc. of ACM SIGGRAPH. (2000)
145–156
11. Goesele, M., Lensch, H.P., Lang, J., Fuchs, C., Seidel, H.P.: Disco: acquisition oftranslucent objects. In: Proc. of ACM SIGGRAPH. (2004) 835–844
12. Sen, P., Chen, B., Garg, G., Marschner, S., Horowitz, M., Levoy, M., Lensch, H.:
Dual photography. Proc. of ACM SIGGRAPH (2005) 745–755
13. Peers, P., Mahajan, D.K., Lamond, B., Ghosh, A., Matusik, W., Ramamoorthi, R.,
Debevec, P.: Compressive light transport sensing. ACM Trans. Graph. 28 (2009)
1–18
14. Yu, Y., Debevec, P., Malik, J., Hawkins, T.: Inverse global illumination: Recoveringreflectance models of real scenes from photographs. In: Proc. of ACM SIGGRAPH.
(1999) 215–224
15. Boyer, K.L., Kak, A.C.: Color-coded structured light for rapid active ranging.
IEEE Trans. on Pattern Analysis and Machine Intelligence 9 (1987) 14–28
16. Harley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Second
edn. (2004)
17. Lambert, J.H.: Photometry, or, on the measure and gradations of light, colors,and shade: Translation from the latin of photometria, sive, de mensura et gradibus
luminis, colorum et umbrae, 1760. Illuminating Eng. Soc. of North America (2001)
18. Schröder, P., Hanrahan, P.: On the form factor between two polygons. Proc. of
ACM SIGGRAPH (1993) 163–164
19. Kajiya, J.T.: The rendering equation. Proc. of ACM SIGGRAPH (1986) 143–150
20. Arvo, J., Torrance, K., Smiths, B.: A framework for the analysis of error in global
illumination algorithms. Proc. of ACM SIGGRAPH (1994) 75–84
21. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. on
Pattern Analysis and Machine Intelligence 22 (2000) 888–905