A Theory of Inverse Light Transport
Steven M. Seitz
University of Washington

Yasuyuki Matsushita
Microsoft Research Asia

Kiriakos N. Kutulakos∗
University of Toronto

Abstract
In this paper we consider the problem of computing andremoving interreflections in photographs of real scenes. Towards this end, we introduce the problem of inverse lighttransport—given a photograph of an unknown scene, decompose it into a sum of n-bounce images, where each image records the contribution of light that bounces exactly
n times before reaching the camera. We prove the existence of a set of interreflection cancelation operators thatenable computing each n-bounce image by multiplying thephotograph by a matrix. This matrix is derived from a setof “impulse images” obtained by probing the scene with anarrow beam of light. The operators work under unknown
and arbitrary illumination, and exist for scenes that have arbitrary spatially-varying BRDFs. We derive a closed-form
expression for these operators in the Lambertian case andpresent experiments with textured and untextured Lambertian scenes that confirm our theory’s predictions.

1 Introduction
Modeling light transport, the propagation of lightthrough a known 3D environment, is a well-studied problem in computer graphics. However, the inverse light transport problem—using photos of an unknown environment toinfer how light propagates through it—is wide open.
Modeling inverse light transport enables reasoning about
shadows and interreflections–two major unsolved problems
in computer vision. Aside from the interesting theoretical questions of what about these effects can be inferredfrom photographs, understanding shadows and interreflections has major practical importance, as they can accountfor a significant percentage of light in a photograph. Modeling inverse light transport can greatly expand the applicability of a host of computer vision techniques, i.e., photometric stereo, shape from shading, BRDF measurement,
etc., that are otherwise limited to convex objects that do notinterreflect light or cast shadows onto themselves.
The intensities recorded in an image are the result of
a complex sequence of reflections and interreflections, aslight emitted from a source will bounce off of the scene’s
surfaces one or more times before it reaches the camera. In∗ Part of this research was conducted while the author was serving as a
Visiting Scholar at Microsoft Research Asia.

I1

I2

I3

Figure 1: An n-bounce image I n records the light that bounces
exactly n times before reaching the camera.
theory, therefore, every image can be thought of as an infinite sum, I = I 1 + I 2 + I 3 + . . ., where I n records the contribution of light that bounces exactly n times before reaching the camera (Figure 1). For instance, I 1 is the image wewould capture if it were possible to block all indirect illumination from reaching the camera, while the infinite sum
I 2 + I 3 + . . . represents the total contribution of indirect
illumination. While we can capture the composite image I
using a camera, the individual “n-bounce” images are notdirectly measurable.
In this paper we prove the existence of a set of linear
operators that compute the entire sequence of n-bounce images in a photograph of an unknown scene captured under
unknown illumination. These operators, which we call interreflection cancellation operators, exist under very general conditions—the scene can have arbitrary shape and arbitrary BRDF and be illuminated by an arbitrary illumination field. Moreover, we show that in the special case ofscenes with Lambertian reflectance, we can compute these
operators by first collecting a sequence of images in which
the scene is “probed” with a narrow beam of light that slides
over the surface. We emphasize that our procedure requires
acquiring multiple images of a scene as a preprocessing stepin order to cancel interreflected light from any new photograph. The cancellation process is performed simply bymultiplying the new photograph by a matrix derived fromthe pre-acquired images.
By removing the effects of interreflections, these operators can be used to convert photographs into a form moreamenable to processing using existing vision algorithms,
since many techniques do not account for interreflections.
Techniques for simulating interreflections, and otherlight transport effects are well-known in the graphics com-munity (e.g., work on ray tracing [1, 2] and radiosity [1, 3]),and have been studied in the vision community as well[4, 5]. However, relatively little is known about the inverse problem of modeling the effects of interreflections,in images of real scenes. A number of authors [6, 7, 8]
have recently proposed methods to capture the forward lighttransport function from numerous photographs taken under
different illuminations, implicitly taking into account the
global effects of interreflections, shadows, sub-surface scatting, etc. Yu et al. [9] and Machida et al. [10] describe inverse global illumination techniques that model diffuse interreflections to recover diffuse and specular material properties from photographs. In both methods, the geometry andlighting distribution is assumed to be known a priori. Noneof these methods provides a means, however, for measuring and analyzing the effects of interreflections, in an image
where the scene’s shape and the incoming light’s distribution are both unknown.
Although the focus of our work is cancelling interreflections, it is closely related to the shape-from-interreflections,problem. This problem has received very limited attention,
focusing on the case of Lambertian scenes [11, 12] andof color bleeding effects between two differently-colored
facets [13, 14, 15, 16]. Closest to our analysis is the inspiring work of Nayar et al. [11], who demonstrated an iterative
photometric stereo algorithm that accounts for interreflections. Our analysis differs in a number of interesting ways
from that work. Whereas [11] assumed uniform directional
lighting, we place no restriction on the illumination field.
Moreover, in contrast to the iterative approach in [11] for
estimating the light transport equations and analyzing interreflections, we derive closed-form expressions for the interreflection cancellation operators, that need be applied only
once to an image. On the other hand, a disadvantage of ourapproach compared to Nayar’s is that we require many more
input images to model the transport process.
Our approach is based on two key insights. The first one,already known in the graphics literature, is that the forward
propagation of light can be described using linear equations.
We use this fact to show that the mapping from an arbitrary
input image to each n-bounce component can be expressed
as a matrix multiplication. The second is that in the case ofa Lambertian scene, we can compute this matrix from a setof images in which an individual scene point is illuminatedby a narrow beam of light. Intuitively, each such image
can be thought of as a “impulse response” that tells us how
light that hits a single scene point contributes to the indirect
illumination of the rest of the scene.
We start by proving the existence of linear cancellation
operators that compute the n-bounce images under very
general conditions (Section 2). For the case of Lambertian
scenes and a fixed camera viewpoint, we then show how toconstruct these operators from a set of input images (Section 3). Section 4 presents experimental results.

2 Inverting Light Transport
The appearance of a scene can be described as a lightfield Lout (ω yx ), representing radiance as a function of outgoing ray ωxy [17, 18, 19]. While various ray representations are possible, we specify a ray ωxy by its 3D point oforigin x and direction from x to another point y. We restrict x to lie on scene surfaces. Since the set of rays thattouch the scene is 4D (assuming the scene is composed ofa finite set of 2D surfaces), we may speak of a light field as
being a 4D function. To simplify our analysis, we consider
light of a single wavelength.1 Scene illumination may also
be described by a light field Lin (ωxx ) describing the lightemitted from source points x to surface points x. We use
the terms outgoing light field and incident light field to refer
to Lout and Lin , respectively.
An outgoing light field Lout is formed by light from one
or more emitters that hits objects in the scene and gets reflected. Some of this reflected light hits other objects, which
in turn hits other objects, and the process continues until anequilibrium is reached. We can therefore think of Lout as
being composed of two components: light that has bounced
a single time off of a scene point (direct illumination), andlight that has bounced two or more times (indirect illumination), i.e.,yLout (ωxy ) = L1out (ωxy ) + L2,3,...
out (ωx ) .

(1)

The first component, L1out (ωxy ), is determined by the BRDF
of x and the emitters that illuminate x. The second component depends on interreflected light that hits x from otherpoints in the scene. Eq. (1) can be expressed as an integral
equation, known as the light transport equation or the rendering equation [20, 3] in the computer graphics literature:
Lout (ωxy ) = L1out (ωxy ) +A(ωxy , ωxx )Lout (ωxx )dx .
x

(2)
The function A(ωxy , ωxx ) defines the proportion of irradiance from point x to x that gets transported as radiancetowards y. As such, it is a function of the scene’s BRDF,
the relative visibility of x and x and of light attenuation
effects [20, 3].2 When x = x , A(ωxy , ωxx ) is 0.
If we assume that the scene is composed of a collectionof small planar facets and if we discretize the space of rays,
Eq. (2) can be expressed in a discrete form [3, 11] asA[i, j]Lout [j] ,(3)Lout [i] = L1out [i] +jwhere Lout is a discrete light field represented by a setof sampled rays, L1out is the corresponding 1-bounce lightfield, and A[i, i] = 0. Rewriting Eq. (3) as a matrix equation
yields(4)
Lout = L1out + ALout .
1 Color may be treated by considering each wavelength separately.
2 Formally, Eq. (2) interprets the light reflected by x from external lightsources as if it came directly from x, i.e., x is treated as an emitter.

Eq. (4) defines, in a discrete form, how light is transported
through a scene. A direct solution is obtained by solving
Eq. (4) for Lout , obtaining [20, 3]−1

Lout = (I − A)

L1out .

(5)This equation, well known in the graphics community,
shows that the global effects of light transport can be modeled by a linear operator (I − A)−1 that maps a light fieldcontaining only direct illumination to a light field that takes
into account both direct and indirect illumination.

Applying the cancellation operator to an ISF yields
def

t1i = C1 ti ,where t1i is the component of ti due to 1-bounce reflection.
By defining T1 = [t11 t12 . . . t1m ] we get the matrix equationdef

C1 = I − A .

(6)

From Eq. (5), it follows that
L1out = C1 Lout .

(7)

It is therefore possible to cancel the effects of interreflections in a light field by multiplying with a matrix C1 . We
call C1 an interreflection cancellation operator. Hence C1
exists for general BRDFs, and is linear. Note that this result
is a trivial consequence of the rendering equation—while
we do not believe the cancellation operator has been exploited previously in computer vision, its existence is implicit in the derivations of forward light transport [20, 3, 11].
Even though Eq. (7) provides an existence proof of aninterreflection cancellation operator, C1 is defined in termsof shape and BRDF quantities (contained in the entries ofA) instead of image quantities. We now provide an interpretation in terms of image quantities, as follows.
Consider emitting unit radiance along ray i towards thescene (e.g., using a laser beam or a projector). The resulting
light field, which we denote ti , captures the full transport oflight in response to an impulse illumination. We call ti an
Impulse Scatter Function (ISF) .3 Now concatenate all the
ISFs into an ISF matrix T:def

T = [t1 t2 . . . tm ] .

(8)Because T is made up of ISFs , it is possible in principle tomeasure T in the laboratory using controlled illumination.
Although capturing a full dense set of ISFs would be extremely time- and storage-intensive, previous authors have
explored the problem of capturing two-dimensional formsof T [6, 7, 8].
Because light is linear, any light field Lout can be described as a linear combination of ISFs , enabling applications such as synthetic relighting of scenes [6]. In particular,
we can express any outgoing light field as a function of theillumination Lin by
Lout = TLin .
3 It has also been referred to as the impulse response in [7].

(9)T1 = C1 T(11)

C1 = T1 T−1 .

(12)and therefore

2.1 Cancelling Interreflections
Consider the following operator:(10)

Eq. (12) provides an alternative definition of the cancellation operator C1 in terms of image quantities. Intuitively,
applying C1 to a light field Lout has the effect of firstcomputing the scene illumination field (Lin = T−1 Lout )and then re-rendering the scene with a 1-bounce model(L1out = T1 Lin ).
Note that while T is measurable, T1 is generally not.
Hence, the derivations in this section provide only an existence proof for C1 . Also note that Eq. (12) is valid only
when T is invertible. C1 is well-defined even when T is
not invertible, however, since Eqs. (6) and (7) always hold.

2.2 N-bounce Light Fields
Suppose we wanted to compute the contribution of light
due to the second bounce of reflection. More precisely, suppose light from the light source first hits a point p, thenbounces to point q, and then is reflected toward the camera (see image I 2 of Figure 1). How can we measure thecontribution of this light to the image intensity at q’s projection? While this problem has a straightforward solution
when the scene’s shape, BRDF, and illumination are known[1], we are not aware of any solutions to this problem for
unknown shapes and illuminations. Beyond removing interreflections, C1 offers a simple way of solving this problem.
In particular, given a light field Lout of the scene, theportion of light due purely to interreflections is given by
Lout − C1 Lout . Since the direct illumination has been
removed, we can treat the indirect illumination comingfrom each visible point p as if it were coming directlyfrom a light source located at p. Hence, the light field
C1 (Lout − C1 Lout ) is the component of Lout that is due tothe second bounce of light. More generally, the nth -orderinterreflection cancellation operator and the n-bounce lightfield are given by
Cn

def

Lnout

def

=
=

n−1

C1 (I − C1 )andn

C Lout ,where Lnout defines a light field due to the nth bounce oflight. This light has hit exactly n scene points between thelight source and the camera. We can therefore “unroll” theindividual steps of light transport as it propagates throughthe scene by expressing the outgoing light field Lout as asum of individual n-bounce light fields:
Lout =∞


Lnout .

n=1

3 The Lambertian Case
While the results in Section 2 place no restriction on the
BRDF or range of camera viewpoints, they provide only
an existence proof of inverse light transport operators. We
now turn to the problem of computing Ci . To this end, weshow that if the scene is Lambertian, we can compute these
operators from images taken at a single viewpoint.
A defining property of Lambertian scenes is that eachpoint radiates light equally in a hemisphere of directions.
We may therefore reduce the 4D outgoing light field to a
2D set of outgoing beams, one for each point on the surface. A second property of Lambertian scenes is that if we
illuminate a point from two different directions, the patternof outgoing radiance (and hence interreflections) is the same
up to a scale factor. We may therefore reduce the 4D incident light field Lin to a 2D subset.
Because the incident and outgoing light fields are both
2D for the Lambertian case, we can capture an ISF matrix
by scanning a narrow beam of light over the surface andcapturing an image from a fixed camera for each positionof the beam. Each ISF ti is constructed by concatenating
the pixels from the i-th image into a vector and normalizing
to obtain a unit-length vector (thereby eliminating the scaledependence on incoming beam direction).
We now assume without loss of generality that there is a
one-to-one correspondence between m scene points, m image pixels, and m incident light beams, i.e., incident beami hits scene point i which is imaged at pixel i.4
We assume that only points which reflect light (i.e., havepositive albedo) are included among the m scene points.
Finally, to simplify presentation, we assume that all points
that contribute reflected light to the image, direct or indirect, are included among the m points. We relax this lastassumption in Section 3.1.2.
Our formulation of the Lambertian ISF matrix, together
with the interreflection equations, leads to a closed-form
and computable expression for the interreflection operator:
Lambertian Interreflection Cancellation Theorem: Each view of a Lambertian scene defines aunique m × m interreflection cancellation operator, C1 , given by the expression
C1 = T1 T−1 ,(13)where T1 is a diagonal m × m matrix containing
the reciprocals of the diagonal elements of T−1 .
4 This is achieved by including only beams that fall on visible scene
points and removing pixels that are not illuminated by any beam.

Proof: From Eqs. (6) and (12) we have
C1 = T1 T−1 ,where C1 [i, i] = 1, and C1 [i, j] = −A[i, j]. Since only
one point pi is illuminated in each ISF and pi appears onlyat pixel i, it follows that T1 must be a diagonal matrix.
Since T1 is diagonal and C1 has ones along the diagonal, it
follows that1
T1 [i, i] = −1.
T [i, i]
QEDThis closed-form expression for C1 provides explicit information about surface reflectance and relative visibility.
Specifically, from [11] we obtain
C1 = I − PK ,(14)

where P is a diagonal matrix with P[i, i] specifying thealbedo for point i divided by π, and K is the matrix of diffuse form factors [21]. Moreover, Eq. (14) implies that our
method can handle variation in surface texture.
Since T is formed from impulse images, its diagonal elements correspond to points that were directly illuminatedby the laser. Hence, the diagonal of T dominates the otherelements of the matrix. In practice, we have found that T iswell conditioned and easy to invert.

3.1 Practical Consequences
3.1.1 General 4D incident lighting
A key property of the Lambertian cancellation operator
C1 is that it cancels interreflections for the given viewpoint
under any unknown 4D incident light field. Because thespace of ISFs for the Lambertian case is 2D and not 4D,
it follows that any outgoing light field Lout (under any 4D
incident illumination) can be expressed as a linear combination of 2D ISFs . The coefficients of this linear combination determine an equivalent 2D illumination field (along
the m rays defined by the captured ISFs ) that produces alight field identical to Lout .
It follows that C1 will correctly cancel interreflections
even in very challenging cases, including illumination froma flashlight or other non-uniform area source; from a video
projector sending an arbitrary image; or from an unknown
surrounding environment. See Section 4 for demonstrationsof some of these effects.
3.1.2 Interactions with occluded points
To facilitate our proof of the interreflection theorem,
we assumed that an ISF was captured for every point that
contributes reflected light to the image (direct or indirect).
Stated more informally, every point that plays a role in thelight transport process must be visible in the image. This
assumption is not as restrictive as it sounds because ourformulation allows for multi-perspective images—for instance, we can create an “image” by choosing, for every

p2

p1

p2

Image forbeam position i

p1
1

2

3

4

1
2
3

(a)

(b)

Figure 2: (a) Light hitting p1 interreflects off of occluded points
that are not within the field of view of the camera (dotted lines),causing additional light to hit p1 and p2 . (b) An alternative explanation is that there are no occluded points, but additional lightflowed from p1 directly to p2 , and from external light sources to
p1 (dashed lines).

point on the surface of the object, a ray that ensures thepoint is visible.
In practice, it may be difficult to capture an ISF for every
point, and it is convenient to work with single-perspective
images that contain occlusions. It is therefore important
to consider what happens in the case of interactions withoccluded points, i.e., points that are not represented in thecolumns of T. Fortunately, the cancellation theorem also
applies to such cases because of the following observation.
Suppose that light along a beam from a visible point pi
bounces off of k invisible points before hitting the first visible point pj (Figure 2a). We can construct a different interpretation of the same image that does not involve invisible points by treating all of this light as if it went directlyfrom pi to pj , i.e., the k bounces are collapsed into a single bounce (Figure 2b). It is easy to see that the transport
equation, Eq. (3), still applies—there is just more light flowing between pi and pj . The collapsed rays have the effectof (1) increasing A[i, j] to take into account the additional
light that flows “directly” from pi to pj and (2) increasing
the apparent amount of “direct” illumination L1out [i] and,hence, t1i .
It follows that C1 applies as before, with the modification that light which pi sends to itself via any number ofintermediate bounces off of invisible surfaces is treated as
direct illumination and therefore not cancelled. Similarly,
C2 will not cancel light that pi sends to pj through any
number of intermediate bounces off of invisible surfaces.

4 Experimental Results
In order to confirm our theoretical results, we performedexperiments with both real and synthetic scenes. Our emphasis was on computing cancellation operators and nbounce images, and comparing them to ground-truth fromsimulations.
Our experimental system consisted of a Canon EOS20D digital SLR camera, a 50mW collimated green laser
source, and a computer-controlled pan-tilt unit for directing the laser’s beam. To minimize laser speckle, we useda Canon wide-aperture, fixed focal length (85mm) lens andacquired images using the widest possible aperture, F 1.2.

Beam position i

4

Pixel iT

Figure 3: Graphical representation of the ISF matrix for the syn-thetic “M” scene. The image captured at the i-th beam position
becomes the i-th column of the ISF matrix T. The pixel receiving direct illumination (position i) is mapped to element T(i, i).
Because the scene contains four facets, T has 42 = 16 “blocks:”
the block shown in gray, for instance, describes the appearance ofpoints on facet 2 when the beam is illuminating a point on facet 4.

Figure 4: Left-top: image of the “M” scene. Spots in top-leftimage indicate the pixels receiving direct illumination as the beam
panned from left to right. Left-middle: One of the captured images, corresponding to the 4th position of the laser beam. The 4thcolumn of the ISF was created by collecting the pixels indicatedby the spots at top-left and assembling them into a column vector.
Left-bottom: image of the scene corresponding to another beam
position. Right: The complete ISF matrix.
To exploit the extreme dynamic range offered by the laser(over 145dB), we acquired HDR images with exposuresthat spanned almost the entire range of available shutter
speeds—from 1/15s to 1/8000s with an additional image
with an aperture of F 16 and a speed of 1/8000. All images
were linearized with Mitsunaga et al’s radiometric calibration procedure [22]. We used only the green channel bydemosaicing the raw CCD data.
To capture a scene’s ISF , we moved the laser beam toa predetermined set of m directions and captured a highdynamic range image for each direction. Since our analysis
assumes a known one-to-one correspondence between them illuminated scene points and the pixels they project to,
we first determined the pixel that received direct illumination. To do this, we found the centroid of the laser spot in
the shortest-exposure image for each beam direction.These

1

2

3

4

Simulated

12
3
4
Reaｌ data

C1T

T

C2T

C 3T

C4T

T − C1T

Figure 5: Inverse light transport computed from the ISF matrix T for the “M” scene. Top row shows typical light paths, middle row shows
simulation results, and bottom row shows results from real-world data. From left to right: the ISF matrix, direct illumination (1-bounce),
the 2-, 3- and 4-bounce interreflection components of the ISF , and total indirect illumination. Images are log-scaled for display purposes.
centroids were used to sample all m input images. Hence,each image provided m intensities, corresponding to a column of the m × m ISF matrix.

I
I1

I 2 (× 4 )I 3 (× 4 )I 4 (× 4 )I 5 (× 12 )I − I1Figure 6: Inverse light transport applied to images I captured

under unknown illumination conditions. I is decomposed into
direct illumination I 1 and subsequent n-bounce images I n , asshown. Observe that the interreflections have the effect of increasing brightness in concave (but not convex) junctions of the “M”.
Image intensities are scaled linearly, as indicated.

Computing k-bounce operators in 1D As the first experiment, we simulated a scene containing a Lambertian surface shaped like the letter “M.” To build its ISF matrix, we
simulated an incident beam that moves from left to right(Figure 3). For each position of the beam, we rendered
an image by forward-tracing the light as it bounces fromfacet to facet, for up to 7 bounces. These images formed
the individual columns of the ISF matrix. At the same time,
our renderer computed the contribution of individual lightbounces in order to obtain the “ground truth” n-bounce images for the scene.
The top row of Figure 5 shows the result of decomposing
the simulated ISF matrix into direct, 2- through 4-bounce,and indirect illumination components. To assess this decomposition qualitatively, consider how light will propagate
after it hits a specific facet. Suppose, for instance, that we illuminate a point on the scene’s first facet (red column of the
ISF matrix in Figure 5). Since only one point receives direct illumination, cancelling the interreflections in that column should produce an image with a single non-zero value
located at the point’s projection. The actual cancellation
result, indicated by the red column in C1 T, matches thisprediction. Now, light reflected from facet 1 can only illuminate facets 2 and 4. This implies that the 2-bounce image should contain non-zero responses only for points onthose two facets. Again, application of the 2-bounce operator, C2 T, yields the predicted result. More generally,light that reaches a facet after n bounces cannot illuminatethat facet in the (n + 1)−th bounce. We therefore expect
to see alternating intensity patterns in the 4 × 4 blocks of
Cn T as n ranges from 2 to 4. The computed n-bounce matrices confirm this behavior. These cancellation results arealmost identical to the ground truth rendered n-bounce images, with squared distances between corresponding (normalized) columns that range from 3.45e-34 for the 1-bounce
image to 8.59e-13 for the 3rd bounce.
We repeated the same experiment in our laboratory witha real scene whose shape closely matched the scene in our
simulations (Figure 4). The result of decomposing the captured ISF matrix is shown in the bottom row of Figure 5.
The computed n-bounce matrices are in very good agreement with our simulation results. The main exceptions
are near-diagonal elements in the 2-bounce matrix, C2 T.
These artifacts are due to lens flare in the neighborhood of
the directly-illuminated pixel. Lens flare increases the intensity at neighboring pixels in a way that cannot be explained by interreflections and, as a result, the intensities
due to flare cannot be cancelled by C2 .
Inverting light transport in 1D A key feature of our theoryis that it can predict the contribution of the n-th light bounce
in images taken under unknown and completely arbitrary
illumination. Figure 6 shows the results of an experimentthat tests this predictive power. We took two photos of the“M” scene while illuminating it with a flashlight and withroom lighting. The resulting images are quite complex, exhibiting a variety of soft and sharp shadows. Despite thesecomplexities, the computed n-bounce images successfully
isolate the contribution of direct illumination, whose direction of incidence can be easily deduced from the locationof the sharp shadows. Moreover, the higher-order operators
allow us to track the propagation of light through the sceneeven up to the 5th bounce.
Inverting light transport in 2D To test our theory further,
we computed n-bounce operators for the 2D scene configuration shown in Figure 7. The scene consisted of threeinterior faces of a box. We chose laser beam directions
that allowed us to sample points on all three faces of thescene. Figure 8 shows the scene’s ISF matrix. Each column in this matrix represents the intensity of a 2D set ofimage pixels, in scanline order. The figure also shows thecomputed decomposition of the matrix into direct, indirect,
2-bounce, 3-bounce and 4-bounce of reflections. Figure 9shows results from inverting the light transport process inimages of the scene taken under three different illumination
conditions. While ground truth information for this scene
was not available, the alternating intensity patterns visible
in the computed n-bounce images are consistent with the
expected behavior, in which light that illuminates a specificface after n bounces cannot illuminate it in the next bounce.

Figure 7: The 2D scene and its ISF matrix T. One columnof the ISF matrix represents the resampled image captured
at a corresponding laser point.

5 Conclusions
This paper addressed the problem of computing and removing interreflections in photographs of real scenes. We
proved the existence of operators that cancel interreflections
in photographs when the geometry, surface reflectance, andillumination are all unknown and unconstrained. For thecase of Lambertian scenes we demonstrated that such operators can be computed, and verified the correctness and viability of the theory on both synthetic and real-world scenes.
Problems for future work include devising more efficientmethods for capturing ISF matrices and estimating cancellation operators for non-Lambertian scenes.

Acknowledgements
We thank Steven Gortler for helpful discussions on lighttransport. This work was supported in part by National Science Foundation grants IRI-9875628, IIS-0049095, and IIS0413198, an Office of Naval Research YIP award, the Natural Sciences and Engineering Research Council of Canada
under the RGPIN program, fellowships from the Alfred P.
Sloan Foundation, and Microsoft Corporation.

References[1] J. D. Foley, A. van Dam, S. K. Feiner, and J. F. Hughes,
Computer Graphics: Principles and Practice. Reading, MA:
Addison-Wesley, 1990.
[2] H. W. Jensen, Realistic Image Synthesis Using Photon Mapping. Natick, MA: AK Peters, 2001.
[3] D. S. Immel, M. F. Cohen, and D. P. Greenberg, “A radiosity
method for non-diffuse environments,” Computer Graphics(Proc. SIGGRAPH), vol. 20, no. 4, pp. 133–142, 1986.
[4] B. K. P. Horn, “Understanding image intensities,” Artificial
Intelligence, vol. 8, no. 2, pp. 201–231, 1977.
[5] J. J. Koenderink and A. J. van Doorn, “Geometric modes asa general method to treat diffuse interreflections in radiometry,” J. Opt. Soc. Am. A, vol. 73, no. 6, pp. 843–850, 1983.
[6] P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin,
and M. Sagar, “Acquiring the reflectance field of a humanface,” in Proc. SIGGRAPH, pp. 145–156, 2000.

T

C1T

C2T

C3T

C4T

T − C1T

Figure 8: The inverse light transport computed over the 2D ISF matrix T. From left to right, the ISF matrix, direct
illumination (1-bounce), indirect illumination, the 2-bounce, 3-bounce and 4-bounce of interreflections (log-scaled).
M. Levoy, and H. P. Lensch, “Dual photography,” ACM
Transactions on Graphics (Proc. SIGGRAPH), 2005. to appear.

I

I1

I 2 (× 3)

I 3 (× 3)

I 4 (× 6)

I 5 (×15)

I − I1[9] Y. Yu, P. E. Debevec, J. Malik, and T. Hawkins, “Inverse
global illumination: Recovering reflectance models of realscenes from photographs,” in Proc. SIGGRAPH, pp. 215–
224, 1999.
[10] T. Machida, N. Yokoya, , and H. Takemura, “Surface reflectance modeling of real objects with interreflections,” inProc. Int. Conf. on Computer Vision, pp. 170–177, 2003.
[11] S. K. Nayar, K. Ikeuchi, and T. Kanade, “Shape from interreflections,” Int. J. of Computer Vision, vol. 6, no. 3, pp. 173–
195, 1991.
[12] D. Forsyth and A. Zisserman, “Reflections on shading,”
IEEE Trans. on Pattern Analysis and Machine Intelligence,
vol. 13, no. 7, pp. 671–679, 1991.
[13] S. A. Shafer, T. Kanade, G. J. Klinker, and C. L. Novak, “Physics-based models for early vision by machine,” inProc. SPIE 1250: Perceiving, Measuring, and Using Color,pp. 222–235, 1990.
[14] R. Bajcsy, S. W. Lee, and A. Leonardis, “Color image segmentation with detection of highlights and local illuminationinduced by inter-reflections,” in Proc. Int. Conf. on Pattern
Recognition, pp. 785–790, 1990.
[15] B. V. Funt, M. S. Drew, and J. Ho, “Color constancy frommutual reflection,” Int. J. of Computer Vision, vol. 6, no. 1,
pp. 5–24, 1991.
[16] B. V. Funt and M. S. Drew, “Color space analysis of mutual
illumination,” IEEE Trans. on Pattern Analysis and MachineIntelligence, vol. 15, no. 12, pp. 1319–1326, 1993.
[17] L. McMillan and G. Bishop, “Plenoptic modeling: An
image-based rendering system,” in Proc. SIGGRAPH 95,
pp. 39–46, 1995.
[18] M. Levoy and P. Hanrahan, “Light field rendering,” in Proc.
SIGGRAPH 96, pp. 31–42, 1996.

Figure 9: Inverse light transport applied to images captured
under unknown illumination conditions: input images I aredecomposed into direct illumination I 1 , 2- to 5-bounce images I 2 –I 5 , and indirect illuminations I − I 1 .
[7] M. Goesele, H. P. A. Lensch, J. Lang, C. Fuchs, and H.P. Seidel, “DISCO: acquisition of translucent objects,” ACM
Transactions on Graphics (Proc. SIGGRAPH), vol. 23, no. 3,
pp. 835–844, 2004.
[8] P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz,[19] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen,“The lumigraph,” in Proc. SIGGRAPH 96, pp. 43–54, 1996.
[20] J. T. Kajiya, “The rendering equation,” Computer Graphics(Proc. SIGGRAPH 86), vol. 20, no. 4, pp. 143–150, 1986.
[21] A. S. Glassner, Principles of digital image synthesis. San
Francisco, CA: Morgan Kaufmann Publishers, 1995.
[22] T. Mitsunaga and S. Nayar, “Radiometric self calibration,”in Proc. Computer Vision and Pattern Recognition Conf.,
pp. 374–380, 1999.